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APPENDIX A. ADDITIONAL DETAILS ABOUT COLLEGE TRANSITION MESSAGING 


This appendix provides additional information about the text-message-based advising, including the specific college 
transition messages that the study sent to students and the training it provided to advisors. This detail is intended to 
allow others to use or build upon the messaging program this study tested. 


A.1 Text Messages 


The transition messaging incorporated promising approaches emerging from recent rigorous research investigating 


low-cost ways to address common barriers to college enrollment and persistence (Exhibit A.1). 


Exhibit A.1 


Barrier 


Research-Based Approaches to Address Common Barriers 


Approach 


Lack of information about key milestones 
(informational) 


Students lack information or have trouble digesting 
information about key college milestones. 


Procrastination and limited attention to milestones 
(behavioral) 


Students who are aware of key milestones could 
nonetheless procrastinate and miss important deadlines as 
they juggle academic, work, and family commitments. 


Limited connections to professional advising 
(logistical) 

Even students who know about important milestones and 
have plans to complete key tasks could struggle. Access to 
professional advising might be limited during key stages of 
the academic pipeline, particularly the summer after 
students leave high school and the first year of college 
when they are still learning how to navigate their college 
campus. 


Fixed mindset and feelings of alienation 
(psychological) 

Students could doubt whether they belong in college or 
their own potential to adjust to the demands of college- 
level work. 


Provide semi-customized real-time text reminders about 
key milestones, such as required pre-matriculation tasks 
and financial aid requirements including refiling the Free 
Application for Federal Student Aid (FAFSA) and meeting 
schools’ Satisfactory Academic Progress (SAP) benchmarks 
to maintain aid (Bird and Castleman 2016; Castleman and 
Page 2015; Page, Castleman, and Meyer 2020; Roderick et 
al. 2008; Schudde and Scott-Clayton 2014). 


Provide action-oriented prompts tied to specific deadlines 
(Castleman and Page 2016; Castleman, Schwartz, and 
Baum 2015; Karlan et al. 2010). 


Provide ongoing access to professional advisors who reach 
out to students and share links to on-campus supports 
(Bettinger and Baker 2014; Bettinger et al. 2012; Barr and 
Castleman 2017; Castleman and Page 2015, 2016; 
Castleman, Schwartz, and Baum 2015; Oreopoulos and 
Petronijevic 2016). 


Provide brief reminders that ease students’ stress and 
anxiety about adapting academically and socially to new 
situations (Aronson, Fried, and Good 2002; Walton and 
Cohen 2011; Yeager et al. 2014). 


The GEAR UP transition messaging consisted of 44 text messages adapted from previous promising research with 
the support of leading academic researchers in behavioral economics and social psychology. The purpose of the 
individual messages varied, including addressing a milestone, encouraging an adaptive mindset, or providing a tip 
for success or logistical information about the text messaging. 


e Reminders about key milestones related to enrolling and persisting in college. These messages included: 


o  Action-planning milestone messages that inform students about key upcoming college milestones— 
for example, to create a study plan for approaching midterms or explore options for summer 
internships or work opportunities. 


o Action-taking milestone messages that prompt students to complete a task, such as paying tuition 
by the due date or registering for classes, and that direct students where to go to complete it by 
providing a web link or the name of an on-campus office. 


e Mindset prompts that promote an adaptive mindset by incorporating social psychological concepts from 
research. These messages included: 


o Growth mindset messages to encourage students’ belief that abilities, particularly academic abilities, 
can be developed through effort. For example, these messages reference research showing that the 
brain grows as it is challenged. 


o Social belonging messages to inform students that everyday worries about fitting in at college are 
normal at first, and pass with time. 


e Success strategies offering concrete tips and suggesting campus resources where students could seek 
additional support. Examples of success strategies included taking advantage of professors’ office hours and 
forming a study group to prepare for exams. 


e Texting logistics messages that communicate information related to the implementation of the texting 
program itself, such as the start and end of the program. 


A.1.1 Developing the Messages 


Adapting the Messages from Other Text Messaging Programs 


The text messages for this study were adapted from messages used in prior research conducted by Castleman and 
Page (2015, 2016). Dr. Benjamin Castleman from the University of Virginia was a consultant on this study, and he 
provided guidance and feedback on how to adapt all of the messages for the GEAR UP students. This study also 
added new “mindset” messages that drew on prior research that found promising effects of encouraging students to 
have a growth mindset and to recognize that social belonging struggles are common. Dr. Chris Hulleman of the 
University of Virginia helped to adapt this mindset content for text messaging, a delivery mode not used previously. 
Dr. Hulleman is Director of the Motivate Lab, where he researches how growth mindsets and social belonging relate 
to learning and student success. 


Using Pilots to Structure Text Messages 


In addition to drawing on experts with prior experience in text-message-based advising and in developing a growth 
mindset, the study also solicited student feedback on how to best structure text messages to encourage student 
engagement with the advising. First, prior to sending any text messages, the study team conducted focus groups in 
which high school and college students not participating in the effectiveness study provided feedback on the 
wording and format of specific messages. The study team then conducted an additional round of focus groups, this 
time with students from cohort 1 being sent the text messages. The study team took student feedback into account 
in a number of ways. For example, the study team revised the wording of the messages to include the advisor’s name 
so that students knew who would be responding to their messages; the study team also varied the greeting, from 
“Hi” to “Hey” to “Hi again,” so that the messages seemed less generic. 


In addition to personalization to make the text messages appear more natural and less automated, the structure of 
the text messages varied such that some asked questions that were open-ended, whereas others were closed-ended. 
For example, an open-ended message asked students, “Need info about tuition payments?” whereas a closed-ended 
question asked, “Have you completed the federal financial aid application (also known as FAFSA)? Reply YES or NO.” 
Programming of the closed-ended messages generated structured autoreplies that depended on how students 
responded. For example, for the closed-ended question of “Still going to UMass in the fall? Reply Yes or NO,” if the 
student replied YES, the autoreply was “Great to hear! Thanks for letting me know.” 


The study also incorporated experiences from cohort 1 students into the timing of when messages were sent. In 
general, students received messages between 11 a.m. and 3 p.m. local time and never on holidays. The send dates 
and timeframe were chosen to correspond with advisors’ working hours while ensuring that, if students were ina 
time zone different than their advisor’s, messages did not come unreasonably early in the morning or late in the 
evening. For students in cohort 2, the study team also minimized weekend messages. Findings from cohort 1 drove 
this decision: among cohort 1 students who received messages, most responded soon afterwards, and advisors were 
not available to respond quickly to students’ messages on weekends. 


Personalizing Text Messages to Colleges that Students Planned to Attend 


The study personalized some of the text messages based on where students planned to attend or actually attended 
college. This was a key feature of the text messaging program because prior studies suggested that individuals are 
more likely to take action if the information they receive is relevant to them (Castleman and Page 2015). 
Personalized messages included college-specific dates for new student orientation, start dates for the fall and spring 
terms, start dates for fall and spring final exams, tuition due dates, and state and college FAFSA filing deadlines. 
Personalized messages also contained links to the college’s student account portal, academic calendar, and 
webpages for new student requirements, tuition information, and the college tutoring center. 


College-specific personalizing required knowing which college the students in the text messaging group planned to 
attend in the fall. For that, the study team used (a) students’ responses on a student survey administered at the end 
of their senior year of high school; and (b) updates received from students via text message about their intended or 
actual college. Updates via text message came in response to programmed messages that began at the start of the 
summer and in ad hoc text messages from students to their advisor. The process of obtaining the names of colleges 
the students intended to and did enroll in yielded very accurate data. For 86 percent of students, the college name 
provided via the survey or text message response matched where they actually enrolled (Exhibit A.2). 


The study team then manually looked up Key information about the named colleges, such as financial aid deadlines, 
links to the registrar, and orientation dates, in order to populate the text messages with information relative to each 
student. 


Exhibit A.2 Extent to which College Where Messaged Students Enrolled Matched Information Provided via Survey or 
Text 
Number of 
Enrolled Percentage of 
Students Enrolled Studentsa 
College enrolled matched college information provided 1,622 86 | 
Via survey only 205 ila 
Via text only 613 33 
Via both survey and text 804 43 | 
_ College enrolled matched neither survey nor text information 198 11 | 
No survey or text information provided 60 3 


a Percentages for mutually exclusive categories may not add to 100% due to rounding. 


Pilot Experiments 


To further test different message options with students, the study team conducted a series of four pilot experiments, 
focused on testing message structures that researchers could easily adjust. Across the four experiments, students in 
cohort 1 were randomly assigned to receive one of two versions of certain messages (Exhibit A.3). Response rates 
for each version were compared to determine which version yielded more student response. Specifically, these 
experiments explored student responsiveness to messages with: 


1. Aninfographic and text versus text only; 


2. An open-ended question versus a structured “yes/no” question about a key college milestone; 


3. An open-ended question versus a question asking for a “1 to 10” reply; and 


4. Anopen-ended question versus a structured “yes/no” question with growth mindset content (this 
experiment included two sets of messages of each type). 


Exhibit A.3 Messages from Four Experiments Comparing Student Responsiveness 


Experiment 1: An infographic and text versus text only 


Version 1: Message with infographic Version 2: Message without infographic 


(1/2) Hi [first name]! If you haven't yet, log on to your (1/2) Hi [first name]! Have you already logged on to your college’s 
college’s online account. Colleges post Key info there, online account? Colleges post key info there, including updated 
including updated financial aid. financial aid. 
tit (2/2) To log on: [url_webportal]. Need any tips on navigating the 
MANY COLLEGES SEND LOG-IN site? 


INFORMATION AND URL WITH 
YOUR ACCEPTANCE 


(2/2) To log on: [url_webportal]. I’ve included a pic to 
show you generally how colleges organize their portals. 
Need any tips on navigating the site? 


LOOK FOR THE FOLLOWING TABS 
ON YOUR COLLEGE PORTAL 


FINANCIAL AID 
ORIENTATION 


TUITION 


COURSE REGISTRATION 


Experiment 2: An open-ended question versus a structured “yes/no” question about a key college milestone 


Version 1: Open-ended message Version 2: Structured “yes/no” message 


(1/2) Hi! Now is an important time to finalize financial (1/2) Hi! Have you completed your federal financial aid application 
aid for the fall. (also known as FAFSA)? Reply YES or NO. 


(2/2) Many students find the FAFSA and financial aid 
award letter from colleges confusing at first. Can I Longstreet: 
answer any questions for you? 


GRANTS 
LOANS 
WORK STUDY 


COST OF ATTENDANCE 


MAY BE MORE THAN THE FINANGI 


If YES: Great! That’s a major step to take on the road to college. 
scans Have you already received your financial aid award letter from 
[college abbreviation]? Reply YES or NO. 
If YES: Good to hear. Understanding the difference 
between grants and loans can be tricky. Would you like 
some pointers on how to decode your bill? 
eau If NO: OK. It may be on the way, but I'd check the [college 
abbreviation] online account. Do you want some 
suggestions on how to look into this? 
COST OF ATTENDANCE If NO: (1/2): That’s OK, you can still complete it. Doing the 
MA 8 MORE TK THE MANGAL A YU CIM FAFSA can qualify you for thousands of dollars in grants— 
money you don’t have to pay back! 


(2/2) What questions can I answer about completing the FAFSA? 


WORK STUDY 


Experiment 3: An open-ended question versus a question asking for a “1 to 10” reply 


Version 1: Open-ended message Version 2: Structured message with “1 to 10” reply 


Hi [first name]! Just wanted to touch base and see how Hi [first name]! How are you feeling about transitioning to college in 

you're feeling about transitioning to college in a few a few weeks? Reply 1-10, 1=anxious, 10=excited. 

weeks? If 7 OR ABOVE: Great to hear! Some students go back and forth 
between being excited and nervous. That’s totally normal when 
youre going into a new situation. 
If 4-6: That’s OK, it’s normal to be a bit nervous. Most students 
find over time that their nervousness goes away. Text me if you 
want to talk. 
If 3 OR UNDER: It’s normal to feel nervous. Most students find 
that they feel less nervous over time. Let me know if you want 
to talk. 


Experiment 4: An open-ended question versus a structured “yes/no” question with growth mindset content 


Version 1a: Open-ended message Version 2a: Structured “yes/no” message 


Just wanted to see how your coursework is going. Are Just wanted to see how your coursework is going. Are you finding 
you finding any of your classes particularly challenging? | any of your classes particularly challenging? Reply YES or NO. 


YOUR BRAIN GROWS THE MORE YOUR BRAIN GROWS THE MORE 
YOU CHALLENGE IT. YOU CHALLENGE IT. 


PREPARING FOR MIDTERMS PREPARING FOR MIDTERMS 
1S GOOD FOR YOUR BRAIN. 1S GOOD FOR YOUR BRAIN. 


If YES: (1/2) Many students struggle with college-level courses 
and find it helpful to connect w/ an advisor or tutor. Text me 
for help connecting w/ someone on campus. 
(2/2) Also, here are some concrete study strategies that 
some students find useful: http://bit.ly/1SCdbjj. Text me 
back if I can help w/ study tips. 
If NO: Keep up the good work! Probably on your radar already, 
but midterms may be coming up in a few weeks. Visit 
http://bit.ly/1SCdbijj for great midterm study tips. 


Version 1b: Open-ended message Version 2b: Structured “yes/no” message 


Hi [first name]. How’s the social side of college going? Hi [first name]. How’s the social side of college going? Have you 
Have you made personal connections with anyone on made personal connections with anyone on campus? Reply YES or 
campus? NO. 


If YES: Great to hear. Having a strong friend network and 
campus connections goes a long way to helping students 
succeed in college. 


If NO: (1/2) That’s pretty common for students, actually. It can 
take some time to transition to college. 


(2/2) One way to connect socially is getting involved with 
campus groups and activities. Text back if] can help you 
connect. 


From pilot experiment 1, the study team learned that students were slightly more likely to respond to the text 
version without the infographic (version 2: 14.1 percent) than to the version with the infographic (version 1: 10.9 
percent), but the difference was small (3 percentage points) and not statistically significant.1 Therefore, because 
students in the focus groups had consistently said they enjoyed receiving the infographics, the study team included 
infographics in some additional messages. 


From pilot experiments 2 to 4, the study team learned that students were consistently more likely to respond to the 
closed-ended version than the open-ended version of each message (Exhibit A.4). The difference in students’ 
responsiveness was particularly large for experiment 2, in which 44 percent of students sent the closed-ended 
version of the message (version 2) responded to it versus only 13 percent of students sent the open-ended version 
(version 1).2 These results led the study team to incorporate closed-ended questions when possible into the 
messages for the remainder of the study. 


Exhibit A.4 Student Engagement with Text Messages in Four Experiments 
Percentage of Students Who Responded to 5 
Experiment Message ahi ee aoa p-Value 
Version 1 Version 2 
1 10.90 14.20 -3.29 319 
2 13.27 43.94 -30.70 .000 
3 32.17 37.26 -5.09 272 
4 12.96 16.42 -3.45 335 


Sample Sizes: 
Experiment 1: 230 students sent version 1, 219 students sent version 2 
Experiment 2: 212 students sent version 1, 198 students sent version 2 
Experiment 3: 230 students sent version 1, 212 students sent version 2 
Experiment 4: 216 students sent version 1, 201 students sent version 2 


u From these experiments, the study team also learned that some students’ phone providers did not allow transmission of multimedia 
messaging service (MMS) messages, a capability that was required for the infographics. Because information on which phone providers did 
not allow MMS messages was available, the study team sent students who could not receive MMS messages texts with the same content but 
no infographic for the remainder of the study. 


7 The study team did note for experiment 2, however, that students’ responses to the closed-ended version did not contain as much 
substantive information as their responses to the open-ended version, and that the open-ended version elicited more sentiments (as rated 
using the National Research Council Canada’s Word-Emotion Association Lexicon) in key categories, such as “joy,” despite less 
responsiveness overall. 


A.1.2 Full Set of Programmed Text Messages 


To provide support throughout the students’ transition from high school to college, the study sent messages starting 
in June after high school and continuing until the following May. Across the 12-month transition year, students were 
sent the series of programmed text messages shown in Exhibit A.5.3 


Exhibit A.5 


Date Sent 
Early Junea 


Message Topic 


Introduction to 
texting program 


Programmed GEAR UP Text Messages—Timing, Topic, and Content 


Message Content 
(1/2) Hi [S: first name], this is [P: advisor_name] from [P: program_name]. I will be 
sending you messages with important college info. 


(2/2) Visit http://bitly/2k4fYXY to learn more. You can text this # w/ questions and I'll 
respond to you personally. 


Mid-June Confirmation of Hi, it’s [P: advisor_name] again. Still going to [C: coll_abbrev] in the fall? Reply YES or NO. 
choice of college If YES: Great to hear! Thanks for letting me know. 
If NO: Thanks for letting me know. Do you plan to go to college in the spring? 
Late June College student (1/2) Hi [S: first name]! If you haven’t yet, log on to your college’s online account. Colleges 


online account 


post key info there, including updated financial aid. 


mm. 


MANY COLLEGES SEND LOG-IN 
INFORMATION AND URL WITH 
YOUR ACCEPTANCE 


CREATE A USE LINKS LIKE 
PASSWORD THESE IF YOU NEED 
USING INFO FROM HELP ACCESSING 
YOUR STUDENT YOUR STUDENT 
PACKET PORTAL 


LOOK FOR THE FOLLOWING TABS 
ON YOUR COLLEGE PORTAL 


| COURSE REGISTRATION 
HOOSE CLASS 


$ OR SEE YOUR FALL 


3 GEAR UP grantees could participate in the study in spring 2016 (cohort 1), spring 2017 (cohort 2), or both years. Between the first and 
second cohorts, the study team made small changes to the text messages in response to feedback from cohort 1 participants and changes in 
FAFSA policy. These changes included adding two more school-year messages about financial aid for cohort 2. 


Date Sent Message Topic Message Content 


Late June FAFSA and Hi! Have you completed the federal financial aid application (also known as FAFSA)? 
financial aid letter | Reply YES or NO. 


MAKING SENSE OF YOUR 


FINANCIAL AID 


GRANTS 


WORK STUDY 


COST OF ATTENDANCE 


If YES: Great! That’s a major step to take on the road to college. Have you already 
received your financial aid award letter from your college? Reply YES or NO. 
If YES: Good to hear. Understanding the difference between grants and loans can 
be tricky. Would you like some pointers on how to decode your bill? 
If NO: OK. It may be on the way, but I’d check your college’s online account. Do 
you want some suggestions on how to look into this? 
If NO: That’s OK, you can still do it. Doing the FAFSA can qualify you for thousands of 
dollars in grants—money you don’t have to pay back! Text back if I can help. 


Varied; Late New student (1/2) Hey [S: first_name]! Have you signed up for the [C: coll_abbrev] orientation? 
June? orientation (2/2) Need to register? Visit [C: url_orient] for more information. Text back if I can help. 


Early July Tuition bill Hi there! Your tuition bill may be coming soon. Need info about tuition payments? Visit [C: 
reminder url_tuition]. Text back for suggestions. 
UNDERSTANDING YOUR TUITION BILL 5 TIPS FOR MAKING SENSE 
ar OF YOUR TUITION BILL 
J Ain 
© DOUBLE CHECK ALL CHARGES ON YOUR BILL. 
CHARGES VS. CREDITS 
‘© QUESTIONS ABOUT CHARGES? 
* TUITION © PAYMENTS 
© FEES RECEIVED © ANY FINANCIAL AID MISSING? 
* HOUSING © FINANCIAL AID 
* MEAL PLANS CREDITS 
© LOOK INTO TUITION PAYMENT PLAN 
OPTIONS TO SPREAD OUT WHAT YOU 
OWE EACH MONTH. 
‘© PAY TUITION ON TIME TO AVOID LATE FEES. 


Early July New student (1/2) Hi again! Need help w/ new student requirements like placement tests or housing 
requirements forms at [C: coll_abbrev]? 
(2/2) Visit [C: url_admit] or text back with questions. 


Mid-July Feelings about Hi [S: first name]! How are you feeling about transitioning to college at the end of the 
college transition summer? Reply 1-10, 1=anxious, 10=excited. 

If 7 OR ABOVE: Great to hear! Some students go back and forth between being excited 
and nervous. That’s totally normal when you're going into a new situation. 
If 4-6: That’s OK, it’s normal to be a bit nervous. Most students find over time that 
their nervousness goes away. Text me if you want to talk. 
If 3 OR BELOW: It’s normal to feel nervous. Most students find that they feel less 
nervous over time. Let me know if you want to talk. 


Mid-July Financial aid Hi [S: first_ name]! Does your college plan feel affordable? Loan options and financial aid 
options can be confusing. Text back with any questions. 


Varied; Late Tuition bill due Hi [S: first name]. Your tuition bill is due [C: tuit_duedate]. Need info about tuition 
July date payment options? Visit [C: url_tuition]. Text back for help. 


Date Sent NW STM We) 0) (a Message Content 


Late Julyac College Hey [S: first_name], this is [P: advisor_name]. Just checking: Are you still planning to enroll 
enrollment plans in college this fall? Reply YES or NO. 
If YES: That’s great! Where will you be going? 
If NO: Thanks for letting me know. Do you plan to go to college in the spring? 


Varied; Early Travel to college (1/2) Hi. Right now is the time to finalize your travel plans to college. Your school 
August calendar with important dates is here: [C: url_schcal]. 
(2/2) Have you run into any last minute issues I can help you with? 


Varied; Early Feelings about the | (1/2) Hi [S: first_name]! Can you believe the first day of classes at [C: coll_abbrev] is [C: 
August start of classes fall_start]? 
(2/2) How are you feeling about starting up? Reply 1-10, 1=nervous and 10=excited. 
If 7 OR ABOVE: Great to hear! I hope the semester gets off to a great start. I'll write 
during the year and you can always text me for help. Good luck! 
If 4-6: That’s OK, it’s common to feel nervous. These feelings usually fade over time. 
I'll check in over the semester and am here if you need help or want to talk. 
If 3 OR BELOW: That’s actually very normal. It can take time to feel comfortable at a 
new place. I'll check in over the semester. Would it help to talk? 


Late August Introduction to (1/2) Hi, it’s [P: advisor_name]. In addition to messages about important milestones, I’ll 
GEAR UP tips send tips about concrete college success strategies. 
(2/2) I'll send one or two tips a month, and they’ll start with “GEAR UP TIP.” Let me know 
if they’re helpful! 


Early Start of classes Hi [S: first name]. Have college classes started or are they starting soon? Reply YES or NO. 
September?¢ If YES: Good luck with the start of college! Text back to let me know what college you 
are attending. 
If NO: Are you still planning on going to college this fall? 


Varied; Early Encourage growth | (1/2) Hi there. Did you know that believing that you can get smarter matters for how well 
September? mindset you do in school? 


FIXED vs GROWTH MINDSET 


FIXED MINDSET GROWTH MINDSET 
* 1 CAN'T GET © 1 CAN GET BETTER, 
BETTER. 


© FM JUST NOT * TE NEED TO WORK 
GOOD AT THIS ON THIS. 

© IFLHAVETO TRY, © THE WAY TO 
| MUST NOT BE IMPROVE IS TO 
‘SMART KEEP TRYING. 


© MISTAKES MAKE © MISTAKES HELP 
ME LOOK BAD ME LEARN. 


(2/2) How’re you feeling about the start of the college term? Reply 1-10, 1=not so good, 
10=great. 
If 7 OR ABOVE: Great to hear! Really glad things are getting off to a good start. Feel 
free to write back if anything comes up. 
If 4-6: That’s OK, the transition to college takes time for most students. Do you feel 
like you're able to stay on top of your coursework? Reply YES or NO. 
If YES: Good to hear. Joining campus groups can ease the transition. Try Googling 
your college’s student activities office or searching Facebook. Text for more 
ideas. 
If NO: At first, many students are uncertain how they'll do in college. I suggest 
meeting with your advisor or the tutoring center. Text back if I can help. 
If 3 OR BELOW: It’s normal for the transition to college to take some time. What’s 
feeling most challenging? 


Mid-September | Tutoring center GEAR UP TIP: You're not alone when it comes to figuring out assignments. Working w/ 
tutors can help—have you visited the tutoring center? [C: url_tutor] 


Date Sent 
Mid-September 


Late 
September 


Late 
September 


Early October 
Early October Study schedule GEAR UP TIP: Are midterms coming up in the next couple weeks? Writing out a study 
schedule can help you stay on track to prepare for your exams/papers. 


Mid-Octoberé 


Mid-October 


Early 
November 


Early Personal health 
November 


Mid-November 


NY STM Wa) 0) (a 


Check-in on 
coursework 


Place to study 


Social belonging 


FAFSA refiling 
open 


Encourage growth 
mindset 


Professor office 
hours 


Midterms 


Registration for 
spring classes 


Message Content 


Just wanted to see how your coursework is going. Are you finding any of your classes 
particularly challenging? Reply YES or NO. 


YOUR BRAIN GROWS THE MORE 
YOU CHALLENGE IT. 


STUDYING DIFFICULT COURSE 
MATERIAL IS GOOD FOR YOU BRAIN. 


If YES: Many students struggle w/ college courses and find it helps to talk w/ an 
advisor or tutor. Visit http://bit.ly/2kIC10Q for study tips. Text if ] can help. 


If NO: Keep up the good work! Probably on your radar already, but midterms may be 
coming up ina few weeks. Visit http://bit.ly/2kIC10Q for great midterm study tips. 
(1/2) GEAR UP TIP: Finding a comfortable place to study free from distractions is a key 
success strategy for many students. 
(2/2) Have you tried studying or doing assignments in the library? 


Hi [S: first name]. How’s the social side of college going? Have you made personal 
connections with anyone on campus? Reply YES or NO. 
If YES: Great to hear. Having a strong friend network and campus connections goes a 
long way to helping students succeed in college. 
If NO: That’s actually pretty common for students. Transitioning can take time. 
Joining campus groups and activities can help you connect socially. Text if I can help. 


The 2018-19 FAFSA opened Oct. 1. Complete it early to qualify for as much financial aid 
for next year as possible. Text back if I can help. 


(1/2) Hi there. Your brain grows the more you challenge it, like taking difficult courses or 
learning something new: https://www.khanacademy.org/youcanlearnanything# 

(2/2) Watch this short 90-second video for inspiration. Creating a study plan or working 
w/ a tutor can help you do your best. Text if I can help. 


GEAR UP TIP: Most professors have office hours devoted to meeting with students. This is 
a great way to get questions answered and connect with faculty. 
Hi! Just checking in to see if you've already been through midterms. If so, were you happy 
with how you did? Reply YES or NO. 
If YES: Wonderful! Doing well on your first round of midterms is a real 
accomplishment, and sets you up well for the rest of the semester. Keep it up! 


If NO: That’s OK. Adjusting to college takes time. It may be about adjusting study 
strategies. We can work on developing study skills. Text me to find time to connect. 


GEAR UP TIP: Staying healthy is just as important as studying to do well in college. Taking 
time to exercise can help you do your best in the classroom. 


Hi [S: first name]. At many colleges this is when students start to register for classes for 
next term. Have you already registered? Reply YES or NO. 
If YES: Great to hear! Text me back if it would be helpful to discuss your course 
choices. 
If NO: OK. Feel free to text me back if I can help you explore course options or figure 
out the registration process at your college. 


Late November | Studying with GEAR UP TIP: Working with study groups is a great way to cover a lot of course material. 
peers Are there classmates you can study with for finals? 
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Date Sent NW STM Wa) 0) (a Message Content 


Late Encourage growth | (1/2) As finals approach, it’s a good time to remember that your hard work can make you 
Novembere mindset smarter. Remember this 90-second video on growing your intelligence? 
https://www.khanacademy.org/youcanlearnanything# 
(2/2) You're not in this alone. Advisors and tutors can help you on campus, and you can 
always text me for help. Good luck! 


Mid-December? | Congratulations Congrats on your first term of college! I hope you enjoy the winter break. I’ll send more 
on first term texts in the new year. 


Early January Start of spring Happy new year [S: first_name], it’s [P: advisor_name]. Are you looking forward to the 
term start of the new term? Reply YES or NO. 

If YES: Great! You’re already that much closer to your college degree. I hope the term 

gets off to a smooth start. 

If NO: Sorry to hear that. Are you planning to re-enroll? Reply YES or NO. 
If YES: Good to hear. What in particular are you not looking forward to about the 
start of the new term? 
If NO: Thanks for letting me know. Do you want to explore the pros and cons of 
taking a break from college? Text me back to set up a time. 


Mid-January4 State grants and (1/2) Hi [S: first_name]. You may qualify for state grants and scholarships for the 2018-19 
scholarships year. 
(2/2) Check out [P: state_fin_aid] for required steps in applying for aid other than 
submitting the FAFSA. Text back for help. 


Varied; Late FAFSA deadline (1/2) Hi [S: first_ name]. Remember, if you get financial aid, you need to refile FAFSA to 
March? renew aid for the 2018-19 school year. 
(2/2) Filing by [C: fafsa_file] may help you qualify for thousands of dollars in additional 
grant aid. Have you already refiled FAFSA? Reply YES or NO. 


REFILE FAFSA TO RENEW AID 
FOR NEXT YEAR 


A 


FAFSA REQUIREMENTS 


IF YOU GET FINANCIAL AID, YOU NEED 
TO REFILE FAFSA TO RENEW AID FOR 
NEXT YEAR. 


WHAT IS YOUR COLLEGE/STATE’S 
FAFSA DEADLINE? 


WATCH YOUR EMAIL: FAFSA MAY EMAIL 


YOU WITH QUESTIONS ABOUT YOUR 
APPLICATION. 


If YES: Great job! Keep an eye on your email, since FAFSA may email you with 
questions about the info you provided on the application. 
If NO: That’s OK, you still have time. To get started, visit https://www.fafsa.gov. Can I 
help you refile the FAFSA? 
Early February | Satisfactory (1/2) Hi there. If you got financial aid this year, you need to meet Satisfactory Academic 
Academic Progress (SAP), a minimum GPA, to keep aid for next year. 


Progress (2/2) Some students find this process tricky. You can ask your financial aid office for more 
info, or text me back and we can look this up together. 


Mid-February Summer plans (1/2) Hi [S: first name]. Now's the perfect time to get a jump start on great summer 
internship, travel, or work opportunities. 
(2/2) Try Googling your college’s summer learning or career opportunities office, or text 
me back if you want to explore options. 


Late February Midterm study (1/2) Hi. As midterms approach, remember that studying actually strengthens your brain 
plans and expands what you can learn! 
(2/2): Creating a study plan or working with a tutor can help you do your best. Can I help 
make these connections? 
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Date Sent NW STEM Wa) 0) (a Message Content 
Varied; Late FAFSA deadline (1/2) Hi, it’s [P: advisor_name]. [C: coll_abbrev] gives more aid to students who file by [C: 
March> fafsa_file]. 
(2/2) Just checking to see if you’ve been able to submit the FAFSA for next year’s financial 
aid. Reply YES or NO. 
If YES: Great. Depending on when you refiled the FAFSA, you may have gotten an 
email from FAFSA about next steps w/ your aid. Text me w/ any questions. 
If NO: There’s still time. Getting your FAFSA in on time can unlock thousands of 
additional dollars in aid. Text me if you need help w/ FAFSA. 


Late March Post-midterm Hi, it’s [P: advisor_name]. Just checking in to see if you’ve already been through midterms. 
mindset If so, how did they go? Reply 1-10, 1=not so good, 10=great. 

If 7 OR ABOVE: That’s fantastic—keep up the great work! 

If 4-6: That’s pretty common—it takes time to adjust to college work, even into the 

spring. Would it help to connect to an academic advisor or tutor? Reply YES or NO. 
If YES: Great. Can I help you connect with a tutor or advisor? 
If NO: OK. Here’s a link with some concrete study strategies that some students 
find useful: http://bit.ly/2kIC10Q. Text me back if I can help w/ study tips. 


If 3 OR BELOW: Many students find that academic advising or tutoring can help them 
catch up. I’m happy to help you connect with your advisor or a tutor. Feel free to text 
me. 


discuss where you can go for 1:1 help after May. 


Mid-April Fall registration Hi [S: first name]. At many colleges this is the time when students start to register for fall 
classes. Have you already registered for the fall? Reply YES or NO. 
If YES: Great to hear! Text me back if it would be helpful to discuss your fall course 
choices. 
If NO: OK. Feel free to text me back if I can help you explore course options or figure 


out the registration process at your college. 


Mid-April Financial award Hi [S: first name]. This is the time of spring when students often get their financial aid 
letters award letters. Have you gotten yours yet? Reply YES or NO. 
If YES: Great. Someone in the financial aid office can help you answer any questions 
you have, or you can text me back w/ any questions. 


If NO: OK. Do you remember when you filed your FAFSA? 
Varied; Mid- Final exams (1/2) Hi [S: first_name]. Finals are approaching for many college students. This is just a 
May» quick reminder that your intelligence grows with effort! 


(2/2): And you're not in this alone. [C: coll_abbrev] advisors and tutors can help, and I’m 
here, too. Good luck! 


Late Maya Final text - (1/2) Congrats on your first year of college! I hope you've found these messages helpful 
Congrats on term and that you've had a great first year. 
(2/2) This is our last text, but if you need help this summer don’t hesitate to contact 


someone at your college. I wish you the best in your college career! 


Late Marcha Upcoming end of (1/2) Hi. Just a quick note that I'll be here to help you through the end of May. 
text messaging (2/2) I'll keep sending you texts through then. Text with any questions, and we can also 


a Texting logistics message—Logistics messages were not counted as part of the 37 programmed messages as these messages were focused on 
managing the texting program rather than on delivering message-based advising content to students. 

b Message was programmed to be sent on a date that was anchored to a specific college’s milestone, such as orientation, the start of classes, or the 
tuition due date, provided that the college-specific date was known. 

¢ Sent to subset of students who had not shared their college enrollment plans. 

4 Message sent only to students in cohort 2 based on feedback from cohort 1 participants and changes in FAFSA policy. 

e Contained embedded video. 

Note: Students who indicated that they would not be starting college until the spring were sent an abbreviated set of messages. 


A.1.3 Advising through Messaging 


The programmed text messages provided support, either by encouraging students to connect to their GEAR UP 
advisors directly (via texting or another mode) or by referring students to supports available on their college 
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campus. Each GEAR UP advisor had his or her own account on the text messaging platform that included only 
students from that advisor’s group(s) and allowed the advisor to respond to replies, questions, and inquiries for help 
from those students. 


In addition to the programmed text messages, advisors could write and send their own text messages to individual 
students or groups of students. The study asked advisors to check their accounts daily, respond promptly to 
messages, and announce when they were typically available, such as Monday to Friday from 9 a.m. to 5 p.m., as well 
as when they would be away for an extended period of time. Advisor support to students spanned the full 12-month 
texting period, from June of students’ senior year of high school through May of their first year in college. 


A.2 Training of Advisors 


The study provided a series of trainings to GEAR UP advisors to help them support students after they leave high 
school and disperse to colleges. A series of four 60-minute webinars trained advisors on using the text messaging 
platform, the content of the text messages, and best practices in college advising using text messages. The webinars 
occurred at strategic time points in order to avoid overloading advisors with information. Two webinars occurred in 
May, before text messaging to students started; the third in August, at the start of students’ first year of college; and 
the fourth during January of students’ first year of college. Information provided in each webinar focused on 
upcoming issues students might face, such as adjusting to college courses or making connections on campus, and 
activities students might need to complete soon. Each advisor received a handbook that included key content from 
the webinars and suggested text responses to common student questions, as well as where to find additional 
information about financial aid and planning, campus support services, and learning mindsets. 


During the first webinar, the provider of the text messaging platform instructed advisors on how to use the platform. 
Advisors could access additional ongoing technical support throughout the text messaging program. A non-profit 
organization that had experience providing text-message-based advising led the training about Key college advising 
topics addressed in the text messages and best practices and strategies for advising via text messages. Throughout 
the study, the study team provided information to advisors about the study timeline and successful communication 
practices identified during the focus groups conducted with students in cohort 1 and through monitoring the 
messaging platform. 


In addition, advisors and staff from GEAR UP grantees participating in both cohort 1 and cohort 2 advising were 
invited to attend an in-person refresher training session, held during the summer between cohort 1 and cohort 2, 
about how to use the texting platform, answer questions, and provide referrals to campus resources in response to 
students’ text messages. 
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APPENDIX B. ADDITIONAL INFORMATION ABOUT HOW THE STUDY WAS DESIGNED 
AND IMPLEMENTED 


This appendix provides details about the study design and its implementation to address the study’s research 
questions. This detail is intended to answer questions about the sample, data sources and measures, analytic 
methods, and the achieved power of the study. 


B.1 Study Design 


The study design aimed to answer the following three key questions: 


e Did the college transition messaging improve college enrollment and persistence rates for GEAR UP 
students? Addressing this central question adds to emerging evidence about the effectiveness of college 
transition messaging in general and, specifically, as a strategy for GEAR UP grantees to provide supports to 
students in the year after high school graduation. 


e Do impacts vary by key student characteristics or the GEAR UP high schools students attended? This 
information could be useful to the GEAR UP program office in providing technical assistance to grantees 
about using text messaging to reach students, or to individual GEAR UP grantees trying to determine 
whether adopting the college transition messaging is appropriate for them. 


e How intensively was the text messaging implemented? Descriptive information about student and 
advisor engagement in the text messaging provides context for understanding the effectiveness of the 
college transition messaging, which could be important to the ongoing development of strategies like the 
one evaluated in this study. 


To answer these questions, the study team recruited GEAR UP grantees and high schools they serve to participate in 
the study. Within each GEAR UP high school, the study randomly assigned students either to receive regular GEAR 
UP services plus the college transition messages or to receive regular GEAR UP services only during the year after 
high school. This section describes how the study team recruited grantees, schools, and students for the study and 
how it randomly assigned students to the grantee treatment or control groups. 


B.1.1 Study Sample 


The study focused on the first group of GEAR UP grantees that planned to extend services to students for an 
additional year after high school. To identify the eligible grantees, the study used the list of GEAR UP grantees 
maintained by the U.S. Department of Education (ED) from the 2011 fiscal year (FY) onward. All 80 GEAR UP 
grantees funded with FY 2011 funds that served high school seniors in either the 2015-16 or 2016-17 school year 
and planned to provide services to students in their first year of college were eligible to participate in the study. Of 
these grantees, 16 volunteered to participate in either spring 2016 (cohort 1), spring 2017 (cohort 2), or both 
years.‘ These 16 grantees served 282 high schools, of which 81 volunteered for the study. Those 81 high schools 
included 11,805 seniors who were GEAR UP participants.® 


The GEAR UP grantees participating in the study were not selected to represent the full set of 80 eligible grantees. 
However, the similarity of the participating grantees to other eligible grantees suggests that results from this study 
could be used to inform decisions on whether or not to implement text-message-based advising across GEAR UP 


4 Three grantees participated in cohort 1 only, 11 participated in cohort 2 only, and two participated in both cohorts. 


2 Not all seniors in the study high schools were GEAR UP participants. GEAR UP schools can choose to serve all students in a grade level; all 
students in a grade level who reside in public housing; or students who particularly need GEAR UP services, such as students eligible for free 
or reduced-price lunch, limited English proficient students, or students from groups that are traditionally underrepresented in 
postsecondary education. 
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grantees. Exhibit B.1 shows that participating grantees were similar to the full set of eligible grantees on important 
dimensions, with one exception: participating grantees were less likely to be located in a rural area than were all 
eligible grantees.°® 


Exhibit B.1 Characteristics of Participating GEAR UP Grantees versus All Eligible Grantees 


Characteristic (% a (%) (%)! 
Female 49.5 49.8 49.4 
White 29.5 29.8 29.4 
Black 19.0 18.6 19.1 
Other 13.3 10.8 14.0 
Hispanic 38.1 40.8 37.4 
Seniors enrolling in college 47.6 51.5 46.7 
Students completing AP course 5) 10.0 6.9 
Students completing IB course 0.8 11 0.8 
Schools in rural location 52.7 39.4 56.3 
Schools eligible for Title I funds 84.5 84.1 84.6 

TOTAL N 80 16 64 


a The joint F-test p-value comparing all eligible grantees to participating grantees on the characteristics shown above is .9751. 
> The joint F-test p-value comparing participating versus non-participating grantees on the characteristics shown above is .9173. 
cA single racial/ethnic composition variable cannot be created at the grantee level because the source for these data (GEAR UP Annual 
Performance Reports in fiscal year 2016) reports on race and ethnicity separately for some grantees. 
4 Title I of the Elementary and Secondary Education Act provides funds to local educational agencies and schools with high numbers or 
percentages of students from low-income families. 
Note: AP is Advanced Placement. IB is International Baccalaureate. 
Sample Sizes: 
Gender: All=80 grantees, Participating=16 grantees, Not participating=64 grantees 
Race and ethnicity: All=62 grantees, Participating=13 grantees, Not participating=49 grantees 
Historical college enrollment rate: All=64 grantees, Participating=12 grantees, Not participating=52 grantees 
AP/IB course taking: All=79 grantees, Participating=15 grantees, Not participating=64 grantees 
Locale: All=76 grantees, Participating=16 grantees, Not participating=60 grantees 
Title I status: All=76 grantees, Participating=16 grantees, Not participating=60 grantees 
Source: GEAR UP Annual Performance Report fiscal year 2016, Common Core of Data 2014-15 and 2015-16. 


It is also important to look at similarities between schools that participated and those from the same grantee that 
did not participate. If the schools that participated differ in student composition, college enrollment rates, or locale 
then the overall results from this study may not be applicable to all schools in these grantees. Schools’ that 
volunteered for the study were similar to that full set of 282 schools in the 16 participating grantees on important 


a 


Due to missing data, the sample sizes varied for different grantee characteristics; sample sizes for each characteristic are included in the 
exhibit notes. 


NI 


Of these schools, 22 participated in cohort 1 only, 54 participated in cohort 2 only, and five participated in both cohorts. 
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dimensions, including historical college enrollment rate, school locale, and eligibility for Title I funds. Participating 
schools had a higher proportion of Hispanic students and a lower proportion of Black, non-Hispanic students than 
did eligible schools that did not volunteer (Exhibit B.2). However, statistical comparisons of the participating schools 
to all schools in participating grantees and to non-participating schools did not reveal significant differences 
between the sets of schools. 


Exhibit B.2 Characteristics of Participating Schools versus All High Schools in Participating Grantees 
— All Schools in = 
Participating Grantees Participating Schools Non-Participating Schools 
Characteristic (%)@ (%) (%)> 
Female 48.4 48.3 48.5 
Hispanic 25.8 35.0 22.0 
White, non-Hispanic 41.1 42.1 40.8 
Black, non-Hispanic 23.2 14.4 26.9 
Other, non-Hispanic 9.8 8.6 10.3 
Seniors enrolling in college 51.6 54.7 50.6 
Schools in rural location 47.1 43.2 48.7 
Schools eligible for Title I funds 75.4 79.0 73.8 
TOTALN 282 81 201 


a The joint F-test p-value comparing all schools in eligible grantees to participating schools on the characteristics shown above is .6753. 
> The joint F-test p-value comparing participating versus non-participating schools on the characteristics shown above is .3583. 
¢ Title I of the Elementary and Secondary Education Act provides funds to local educational agencies and schools with high numbers or 
percentages of students from low-income families. 
Sample Sizes: 

Gender: All=275 schools, Participating=81 schools, Not participating=194 

Race/ethnicity: All=275 schools, Participating=81 schools, Not participating=194 

Historical college enrollment rate: All=140 schools, Participating=35 schools, Not participating=105 

School locale: All=276 schools, Participating=81 schools, Not participating=195 

Title I status: All=276 schools, Participating=81 schools, Not participating=195 
Source: GEAR UP Annual Performance Report fiscal year 2016, Common Core of Data 2014-15 and 2015-16. 


Within the participating schools, only certain students were eligible for the study. To be eligible for the study and be 
randomly assigned, students needed to complete the survey administered near the end of their senior year of high 
school, indicate that they intended to enroll in college in the fall after high school, and provide a cell phone number. 
Taken together, the study criteria ensured that eligible students could be identified before random assignment, and 
that students would be attending college in the fall and had a phone to receive text messages. 


All 11,805 seniors who were GEAR UP participants in the 81 participating schools were surveyed. The survey 
gathered information from the 9,677 students who completed it about college advising services that they received in 
high school and determined their eligibility to participate in the study. Students who completed the survey and 
indicated they intended to enroll in college in the fall after high school were eligible. Of the 9,677 students who 
completed the survey, 8,545 students did intend to enroll. Finally, students had to provide a cell phone number and 
agree to participate in a lottery that would randomly assign them to either receive the regular GEAR UP supports 
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coupled with college transition messages (the “messaged”/treatment group) or to receive regular GEAR UP supports 
only (the “non-messaged”/control group).® 


Of the 8,545 students who completed the survey and intended to enroll in college, 4,803 opted into the lottery? and 
3,752 students did not. Those 4,803 students formed the study sample. Of them, 2,819 were randomly assigned to 
the treatment group and 1,984 to the control group (Exhibit B.3; see section B.1.2 for additional details on the 
random assignment). 


Exhibit B.3 Flow from Recruitment of Grantees and Schools to Student Study Sample 


Did Not Complete Survey: 
2,128 


Not College-Intending: 
1,121 


High school ee ee | 

seniors in 

participating Opted Out: 
schools: 3,742 


11,805 


Control: 
Did Complete Survey: College-Intending: Opted In: 1,984 
9,677* 8,545 4,803 

Treatment: 

2,819 


All GEAR UP Survey 


College Random 
students 


completion intentions a 
ou 


* 11 students were not eligible for the study because they did not complete the survey before the lottery. 


Because students had to opt in to participate in the study, the participating students may not represent the full set of 
eligible students. Specifically, the students who opted to participate in the lottery are not similar to all eligible 
students nor to eligible students who did not opt into the lottery on the characteristics shown in Exhibit B.4. Because 
of the differences, the study’s results may not be applicable to students who differ from the students included in this 
study. 


Exhibit B.4 Characteristics of Eligible Students Who Opted into the Lottery versus Eligible Students Who Did Not Opt In 


Eligible Students Eligible Students 
All Eligiblea Who Opted into Who Opted out of 
Students the Lottery the Lottery 
Characteristic (%)> (%) (%)¢ 
Gender 
Female 54.7 58.9 49.2 
Race/Ethnicity 
Hispanic 45.9 50.7 BOOMs 
White, non-Hispanic 26.2 19.9 34.4 
Black, non-Hispanic 15.0 16.4 USj.il 
Other, non-Hispanic 12.8 12.9 12.8 
Household Characteristics 
Single-parent household 31.8 32.8 30.5 
First generation to college 67.4 71.4 62.1 


8 Eleven students who completed the survey after the lottery were excluded from the study. 


9 Of them, 876 students participated in cohort 1 (school year 2015-16) and 3,927 students in cohort 2 (school year 2016-17). 
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Eligible Students Eligible Students 
All Eligiblea Who Opted into Who Opted out of 
Students the Lottery the Lottery 
Characteristic (%)> (%) (%)° 
Academic Achievement 
Taken one or more AP/IB courses 52.3 54.2 49.7 
Unweighted cumulative GPA 
3.67-4.00 (mostly As) 26.5 27.4 25.3 
2.67-3.66 (mostly Bs) 44.5 45.3 43.4 
1.67-2.66 (mostly Cs) Pp. 25.6 29.1 
0.67-1.66 (mostly Ds) 1.7 1.6 1.9 
0.00-0.66 (mostly Fs) 0.2 0.1 0.3 
Expects a bachelor’s degree or higher 77.0 80.3 72.6 
College Enrollment Steps Taken before the Lottery 
Provided college name where student plans to enroll 77.0 76.8 77.2 
Intends to enroll in a two-year college4 42.6 40.1 45.7 
Paid a deposit to a college or received a waiver 26.4 26.5 26.3 
Completed or plans to submit the FAFSA 91.0 93.5 87.8 
TOTAL N 8,545 4,803 3,742 


a Eligible students are those who completed the survey before the lottery, indicated that they intended to enroll in college in the fall after high 
school, and provided a cell phone number. 
5 The joint F-test p-value comparing All eligible students to Eligible students who opted into the lottery on the characteristics shown above is 
.0001. 
¢ The joint F-test p-value comparing Eligible students who opted into the lottery to Eligible students who opted out of the lottery on the 
characteristics shown above is .0001. 
4 Intention to enroll in a two-year college was not included in either of the joint tests because of collinearity with the indicator of the student 
having provided a college name. 
Note: AP is Advanced Placement; FAFSA is Free Application for Federal Student Aid; GPA is grade point average; IB is International Baccalaureate. 
Sample Sizes: 

Gender: All=8,377 students, Opted in=4,726, Did not opt in=3,651 

Race/ethnicity: All=8,364 students, Opted in=4,713, Did not opt in=3,651 

Single-parent household: All=8,384 students, Opted in=4,730, Did not opt in=3,654 

First generation to college: All=7,870 students, Opted in=4,453, Did not opt in=3,417 

Taken AP/IB course: All=8,296 students, Opted in=4,680, Did not opt in=3,616 

GPA: All=7,954 students, Opted in=4,474, Did not opt in=3,480 

Expects bachelor’s degree or higher: All=7,986 students, Opted in=4,533, Did not opt in=3,453 

Provided college name: All=8,545 students, Opted in=4,803, Did not opt in=3,742 

Intends to enroll in two-year college: All=6,516 students, Opted in=3,646, Did not opt in=2,870 

Paid deposit or received waiver: All=8,446 students, Opted in=4,762, Did not opt in=3,684 

Completed or plans to submit FAFSA: All=8,428 students, Opted in=4,750, Did not opt in=3,678 
Source: Student survey 2016 and 2017. 


B.1.2 Random Assignment 


The goal of random assignment was to create treatment and control groups that were similar, before the start of the 
college transition messaging, on characteristics likely to be related to college enrollment and persistence. That way, 
any differences in college enrollment and persistence can be attributed to the college transition messaging rather 
than to initial differences between the groups. This study randomly assigned students who opted into the lottery to 
one of two experimental groups: to receive the regular GEAR UP supports with the addition of college transition 
messages (the “messaged”/treatment group) or to receive regular GEAR UP supports only (the “non- 

messaged” /control group). Random assignment occurred before the start of the college transition messaging, in the 
summer of students’ senior year of high school (June 2016 for cohort 1, June 2017 for cohort 2). To help ensure that 
treatment and control group students were similar on key school characteristics and in terms of the regular GEAR 


UP supports and advising they received, the study team grouped students into random assignment blocks based on 
cohort and school. Within each block, the team randomly assigned students to an experimental group. !° 


The study offered schools in each cohort the option to serve as many students as possible through the college 
transition messaging. Schools had the option of giving students a 50 percent chance of being assigned to the 
treatment group (a 1-to-1 treatment-to-control ratio) or a higher chance (67 percent) of being assigned to the 
treatment group (a 2-to-1 treatment-to-control ratio).11 When there were multiple siblings who opted into the 
study, one sibling was randomized and the other sibling(s) was assigned to the same group, to minimize the 
possibility that a sibling assigned to the control group might be exposed to the text messaging if his or her sibling 
was assigned to the treatment group. !2 


The random assignment procedures resulted in treatment and control groups comprising students with similar 
characteristics before the lottery (Exhibit B.5). Two minor differences in the characteristics of the groups are that 
the treatment group students were more likely to be Hispanic and less likely to have taken one or more AP or IB 
courses. However, this small set of differences is about what would be expected to occur by chance, given the 
number of characteristics examined. Even so, these differences were taken into account in the statistical models that 
estimate the impact of the college transition messaging. !3 


Exhibit B.5 Characteristics of Students in Each Group before the Lottery 


Treatment Control Group 
Group Students Students Estimated 
Characteristic (%) (%) Difference p-Value@ 
Female 58.9 58.9 0.1 .962 
Hispanic 48.6 46.1 2.5 023 
White, non-Hispanic 21.0 22:2. -1.2 .208 
Black, non-Hispanic 17.9 18.9 -1.0 184 
Other, non-Hispanic 12.4 12.7 -0.3 .749 


10 College transition text messages purposefully did not begin until after the end of high school. Messages began in June, after high school 
ended, minimizing the extent to which students in the control group might be exposed to the text messages via peers from their high school 
who were assigned to the treatment group. This timing was chosen to minimize spillover to the extent possible. To further minimize the 
chance that control group students had access to the treatment, at the point of recruitment, grantees agreed to refrain from systematically 
texting students in the control group. However, it is possible that close peers from the same high school who were assigned to different 
experimental groups (treatment versus control) could have communicated during the text messaging and that the treatment group student 
might have shared with the control group student some of the information or strategies received via the text messaging. 


11 Of the 81 schools, 27 requested a 2-to-1 treatment-to-control assignment ratio; the other 54 schools used the 1-to-1 assignment ratio. To 
ensure that the analysis accounted for the differing probability of randomization, preserving the integrity of the random assignment, the 
study team included an indicator (or dummy) variable for each randomization block (based on cohort and school). The Institute of 
Education Sciences’ What Works Clearinghouse” considers this approach an acceptable way to account for the differing probability of 
randomization. 


12 This only occurred for one set of siblings in the same school and cohort. 


13 The Institute of Education Sciences’ What Works Clearinghouse” considers this approach an acceptable way to establish baseline equivalence 
on characteristics for which the mean difference between the treatment and control groups is less than .25 standard deviations. In this 
study, differences between the treatment and control groups on all measured characteristics were less than .25 standard deviations. 
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Household Characteristics 


Single-parent household 32.9 33.1 -0.2 913 
First generation to college 71.4 69.0 2.4 .070 
Taken one or more AP/IB courses 52.1 55.0 =) .038 
Unweighted cumulative GPA 2.99 3.00 0.0 562 
Expects a bachelor’s degree or higher IL 81.3 -2.1 .082 
Provided college name where student plans to VID 78.1 -0.6 588 
enroll 
Intends to enroll in a two-year college 39.6 38.2 14 350 
Paid a deposit to a college or received a waiver 27.4 28.0 -0.6 .646 
Completed or plans to submit the FAFSA 93.4 92.9 0.4 535 


a The joint F-test p-value comparing treatment versus control students on the characteristics shown above is .0968. 
Notes: AP is Advanced Placement; FAFSA is Free Application for Federal Student Aid; GPA is grade point average; IB is International 
Baccalaureate. Treatment group (“messaged”) students received the regular GEAR UP supports with the addition of college transition messages; 
control group (“non-messaged”) students received regular GEAR UP supports only. 
Sample Sizes: 

Gender: Treatment group=2,781 students, Control group=1,945 students 

Race/ethnicity: Treatment group=2,775 students, Control group=1,938 students 

Single-parent household: Treatment group=2,783 students, Control group=1,947 students 

First generation to college: Treatment group=2,613 students, Control group=1,840 students 

Taken AP/IB course: Treatment group=2,755 students, Control group=1,925 students 

GPA: Treatment group=2,633 students, Control group=1,841 students 

Expects bachelor’s degree or higher: Treatment group=2,658 students, Control group=1,875 students 

Provided college name: Treatment group=2,819 students, Control group=1,984 students 

Intends to enroll in two-year college: Treatment group=2,120 students, Control group=1,526 students 

Paid deposit or received waiver: Treatment group=2,800 students, Control group=1,962 students 

Completed or plans to submit FAFSA: Treatment group=2,793 students, Control group=1,957 students 
Source: Student survey 2016 and 2017. 


B.2 Data Collection 


The study team collected data from several sources to assess the effects of the college transition messaging and 
describe its implementation. This section first presents the timing of data collection and then details about the data 
sources used to create the study measures. 


B.2.1 Timing of Data Collection for the Study 


As Exhibit B.6 illustrates, data were collected starting in students’ senior year of high school through the start of 
their second year of college. 


Exhibit B.6 Timing of Data Collection, Lottery, and College Transition Messaging 


Q Data Collection Activities 


Collect student rosters from Collect data on Collect data on any Collect data on continuous 
participating high schools immediate college college enrollment enrollment and FAFSA completion 
enrollment and continuous for the second year 
Collect student surveys college enrollment 
Senior Year of First Year of Second Year of 
High School College? College 
Winter | Spring Summer Fall | Winter | Spring Summer Fall | Winter | Spring 


Lottery assigning eligible (2 College transition 
students to receive text VX messaging (treatment) 


messaging or not 


a Assumes immediate college enrollment in the fall after high school. 


B.2.2 Data Sources Used to Create Study Measures 


Data come from a student survey conducted as part of this study, as well as from administrative data maintained by 
ED and other sources. Exhibit B.7 details each of the study’s data sources. 


Exhibit B.7 Data Sources, including the Sample, Timing of Data Collection, Response or Coverage Rate, and Content 
Used for Each 


Student survey 11,805 high e Spring 2016 (cohort 1) 82% Students’ college intentions 
school seniors ° Spring 2017 (cohort 2) (9,667)> and expectations, academic 
participating in background, demographic 
GEAR UP characteristics, and willingness 

to participate in lottery 
| Administrative and National Data 

Rosters of high school 81 high e Winter 2015-16 (cohort1) 100% (81) Records of students who 

seniors participatingin _ schools © Winter 2016-17 (cohort would be rising seniors 

GEAR UP 2)e participating in GEAR UP 

GEAR UP Annual 80 grantees e School Year 2014-15 and 100% (80) Reports submitted by each 

Performance Reports Fiscal Year 2016 GEAR UP grantee, which 

(APRs) include an entry for all 

participating schools 

Common Core of Data 282 high e School Years 2014-15and 94% (265) Data on high school 

(CCD) schools¢ 2015-16 characteristics such as Title I 

status and locale 

Integrated 529 colleges! e School Year 2015-16 97% (512) Data on college characteristics 

Postsecondary 

Education Data System 

(IPEDS) 
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Response 


Data Source Sample Timing of Data Collection Rate (N) Data Obtained 

National Student 4,803 students e Fall2016,2017(cohort1) 8 Student-level information on 
Clearinghouse (NSC) e Fall 2017, 2018 (cohort 2) college enrollment (NSC) and 
and Federal Student Aid receipt of federal student aid 
(FSA) office documenting college 


enrollment, and college to 
where financial aid was sent 


(FSA) 
Federal Student Aid 4,803 students e Fall 2017, 2018, 2019 76% Student-level information on 
(FSA) office (cohort 1) (3,649)h FAFSA completion 
e Fall 2017, 2018, 2019 
(cohort 2) 
Study’s program 2,819 students e School Year 2016-17 100% (2,819 Records of advisors’ 
monitoring system 16 grantees (cohort 1) students) attendance at training 
SMESchoalbcamZOlane 100% (16 webinars and of messages sent 
(cohort 2) grantees) by and to students and 
advisors 


a The study survey can be found at https://www.reginfo.gov/public/do/PRAViewIC?ref nbr=201503-1850-002&8icID=215771. 

> The response rate for cohort 1 was 91% (1,784 students) and for cohort 2 was 80% (7,893 students). 

¢ Five high schools participated in both cohorts. 

4 Data were collected for all eligible grantees across both cohorts. These data were used to compare participating grantees to all eligible grantees 
and to non-participating grantees (see Exhibit B.1). 

e Data were collected for all high schools in participating grantees. These data were used to compare participating high schools to all schools and 
to non-participating schools (see Exhibit B.2). 

f Data were collected for all colleges in which students planned to enroll (as reported on the survey). These data were used to determine whether 
students intended to enroll in a two-year college (see Exhibit B.4). 

8 Of the 4,803 students in the study sample, records were found in both NSC and FSA for 42.3% of students (2,031), in NSC only for 19.1% of 
students (917), and in FSA only for 4.9% of students (235). Students who do not have records (33.7%, or 1,620 students) do not have missing 
values for variables constructed from this data source, because students without records are classified as not having achieved the outcome—for 
example, not enrolling in college. 

4 Percentage (and number) of students with records in the data source. Students who do not have records do not have missing values for variables 
constructed from this data source, because students without records are classified as not having achieved the outcome—for example, not 
completing the FAFSA. 


B.3. Analytic Methods 


The study created measures about students’ college going and estimated the effectiveness of college transition 
messaging on these measures of college going for all students and for particular groups of students and schools. This 
section describes the approach for examining the effectiveness of the study’s college transition messaging. First, it 
describes the rationale for selecting the study measures used to examine the effects of the study’s college transition 
messaging and how these and other study measures were constructed. Measures were constructed to capture (a) 
students’ college going; (b) student, school, and grantee characteristics before the study started; and (c) 
implementation of the college transition messaging and college advising that students received in high school. 
Second, the section provides details on the study’s analytic methods, including the samples and methods used to 
estimate effects of the college transition messaging on students’ college going measures. The final section provides 
the study-achieved power for each measure. 


B.3.1 Study Measures 


Student College Going Measures 


The study examined the effects of the college transition messaging on multiple measures of college going (Exhibit 
B.8). Study measures most directly related to the text messages are discussed in the main body of the report and 
make up the “main” measures. Two of the main measures focus on college enrollment: immediate college enrollment 
after high school and any college enrollment in the year after high school. The study also examined two main 
measures of college persistence: continuous college enrollment in the year after high school and continuous 
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enrollment into the second fall after high school. In addition, because securing financial aid is likely to be an 
essential step for GEAR UP students to persist in college, the study also examined students’ FAFSA completion. This 
appendix examines additional related “exploratory” measures. 


Exhibit B.8 


Challenge to College Enrollment and 
Persistence 


Rationale for Study Measures Examined 


How Text Messages Target that 
Challenge 


Study Measure Used to Test the 
Effectiveness of College Transition 
Messaging 


During the summer after high school, 
students must complete multiple pre- 
college steps such as paying deposits, 
registering for classes, filling out 
housing and health insurance forms, 
and signing up for a college account. 
Students are often without support to 
complete these necessary steps, as high 
school services have ended and college 
supports have yet to begin. 


Students might enroll in college may fail 
to return for a second year. 


One reason students might not continue 
into a second year of college is difficulty 
funding their education. Renewing the 
FAFSA provides access to various types 
of financial aid. 


Text messages starting at the end of 
high school included reminders about 
pre-college steps such as paying the 
tuition bill, attending orientation, taking 
placement tests, and completing 
housing forms. 


Students who did not enroll in college 
immediately after high school were sent 
a modified set of messages in the fall 
that encouraged and supported 
enrollment in the second semester. 


Text messages during the first year 
after high school focused on keeping 
students in college by directing them to 
on-campus resources, reminding them 
of deadlines throughout the year, and 
assuring them that struggles with social 
belonging and fitting in are normal. 


Text messages during the first year of 
college encouraged students to renew 
their FAFSA. 


Immediate college enrollment after high 
school (main measure) 


Any college enrollment during year 
after high school (exploratory 
measure) 


Continuous college enrollment during 
year after high school (main measure) 


Continuous college enrollment into the 
second fall after high school (main 
measure) 


College enrollment in the second fall 
after high school, regardless of any 
prior enrollment (exploratory 
measure) 


FAFSA completion by October 1, the 
second fall after high school (main 
measure) 


FAFSA completion by October 1, the 
first fall after high school (exploratory 
measure)@ 


@ The study explored four additional measures of FAFSA completion (that is, submission without being rejected) and submission (including those 
rejected): (1) FAFSA completion by the second fall after high school among students who were enrolled in year 1; (2) FAFSA submission by the 
first fall after high school; (3) FAFSA submission by the second fall after high school; and (4) FAFSA submission by the second fall after high 
school among students who were enrolled in year 1. Findings are not reported; however, they were consistent with reported findings. 


No bias was introduced into the estimate of the effect of college transition messaging because there were no missing 
data for any of the outcome measures. For all outcome measures, students who had a record of the outcome 
occurring, such as enrolling in college immediately after high school, were classified as achieving the outcome, 
whereas students for whom there was no recorded occurrence were classified as not having achieved the outcome. 
As a result, the full randomized sample was used to analyze the impact of the college transition messaging on all 
outcomes. Exhibit B.9 provides information on how each of the outcome measures was constructed. 
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Exhibit B.9 


Measure 


Immediate college enrollment after high 
school (main measure) 


Any college enrollment during year after 
high school (exploratory measure) 


Continuous college enrollment during year 
after high school (main measure) 


Continuous college enrollment into the 
second fall after high school (main 
measure) 


College enrollment in the second fall after 
high school, regardless of any prior 
enrollment (exploratory measure) 


FAFSA completion by October 1 the second 
fall after high school (main measure) 


after high school (exploratory measure) 


FAFSA completion by October 1 the first fall 


Data Source 


aaa £22 


NSC and FSA 


NSC and FSA 


NSC and FSA 


NSC and FSA 


NSC and FSA 


FSA 


FSA 


Outcome Measures Examined: Data Source and Measure Construction 


Definition/Coding 


1 = Enrollment is documented in any college on October 1, 2016 (cohort 1) or October 1, 2017 (cohort 2) 
0 = Enrollment is not documented in any college on October 1, 2016 (cohort 1) or October 1, 2017 (cohort 2) 


1 = Enrollment is documented in any college at any point between July 1, 2016, and June 30, 2017 (cohort 1) or July 
1, 2017, and June 30, 2018 (cohort 2) 

0 = Enrollment is not documented in any college between July 1, 2016, and June 30, 2017 (cohort 1) or July 1, 2017, 
and June 30, 2018 (cohort 2) 


1 = Enrollment is documented in any college between July 1, 2016, and June 30, 2017 (cohort 1) or July 1, 2017, and 
June 30, 2018 (cohort 2) with any break in enrollment of less than 5 consecutive months during this period 
and/or student is enrolled on October 1, 2016 (cohort 1) or October 1, 2017 (cohort 2) and graduates from 
college after October 1 

0 = Enrollment is not documented in any college between July 1, 2016, and June 30, 2017 (cohort 1) or July 1, 2017, 
and June 30, 2018 (cohort 2) with any break in enrollment of less than 5 consecutive months during this period 
and/or student is enrolled on October 1, 2016 (cohort 1) or October 1, 2017 (cohort 2) and does not graduate 
from college after October 1 


1 = Continuous college enrollment during year after high school and enrollment is documented in a college on 
October 1, 2017 (cohort 1) or October 1, 2018 (cohort 2) 

0 = Continuous college enrollment during year after high school is not documented or enrollment is not 
documented in a college on October 1, 2017 (cohort 1) or October 1, 2018 (cohort 2) 


1 = Enrollment is documented in any college on October 1, 2017 (cohort 1) or October 1, 2018 (cohort 2) 
0 = No enrollment is documented in any college on October 1, 2017 (cohort 1) or October 1, 2018 (cohort 2) 


1 = Administrative record of student submitting a FAFSA that was not rejected in time to receive aid for fall 2017 
(cohort 1) or fall 2018 (cohort 2) 

0 = No administrative record of student submitting a FAFSA that was not rejected in time to receive aid for fall 
2017 (cohort 1) or fall 2018 (cohort 2) 


1 = Administrative record of student submitting a FAFSA that was not rejected in time to receive aid for fall 2016 
(cohort 1) or fall 2017 (cohort 2) 

0 = No administrative record of student submitting a FAFSA that was not rejected in time to receive aid for fall 
2016 (cohort 1) or fall 2017 (cohort 2) 


Note: FSA is Federal Student Aid office; NSC is National Student Clearinghouse. Treatment group (“messaged” students) received the regular GEAR UP supports with the addition of college transition 
messages; control group (“non-messaged” students) received regular GEAR UP supports only. 
Sample Sizes: 2,819 treatment group students and 1,984 control group students. There are no missing data for any of the outcome measures. 


Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018. 
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Measures Describing Students, Schools, and Grantees 


The study uses characteristics measured before the lottery (called “baseline” characteristics) to describe 
participating students, schools, and grantees. Some of the student baseline characteristics act as covariates in the 
statistical models examining the effects of the college transition messaging to take into account possible existing 
differences between the treatment and control group students and to improve the precision of the estimated 
effects. 14 


Data for all student characteristics come from questions in the student survey, administered in the spring of 
students’ senior year, before the lottery. The study survey had a 100 percent response rate for the students in the 
college transition messaging study because completing the survey was a prerequisite to opting into the lottery and 
participating in the study. However, some students skipped individual survey items, resulting in missing data for 
some of the student characteristics. Exhibit B.10 shows the definition of each student characteristic and the 
percentage of students missing data for it. 


Exhibit B.10 Student Characteristics 


Demographic Characteristics 
Race/ethnicity Question 21 (Are you Hispanic or Latino/Latina?) 1.56 2.32 1.87 
Question 22 (What is your race?) 


1 = Hispanic 
2 = White, non-Hispanic 
3 = Black, non-Hispanic 
4 = Other/multiracial, non-Hispanic 
Gender» Question 20 (What is your gender?) 1.35 1.97 1.60 


1 = Female 
0 = Male 


14 To identify which characteristics to include as covariates in each of the statistical models, the study team used a backward selection with 
forward checking procedure, using a p-value less than .20 as the criterion for covariate inclusion. Research has demonstrated that this 
approach is effective for identifying covariates for inclusion in order to minimize the standard error of the impact estimate (Budtz- 
Jorgensen et al. 2007; Maldonado and Greenland 1993; Price, Goodson, and Stewart 2007). In the backward selection method, all covariates 
are entered in the statistical model. Then the covariate with the largest p-value that is also greater than .20 is dropped from the model. This 
step is repeated until the only covariates remaining in the model meet the p < .20 criterion. In the forward checking step, each of the 
previously eliminated covariates is added back one at a time into the model. Any previously eliminated covariate that has a p-value less than 
.20 when re-introduced is retained in the model. 


Three characteristics were included as covariates in the statistical models regardless of their p-value: gender, potential first-generation 
college student status, and intent to enroll in a two-year college. These covariates were included because research on similar college 
transition programs suggests that effects may differ for groups of students defined by these characteristics. Given the study’s investigation 
of group differences in effects, the study retained these covariates in all statistical models. 
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Socioeconomic Status 


Potential first- Question 23/26 (What is this parent’s or guardian’s 7.31 7.26 7.29 
generation college _ relationship to you?) 
student Question 24/27 (What is the highest level of education he/she 


has completed?) 


1 = Neither parent/guardian received a college degree 

0 = At least one parent/guardian received a college degree 
Single-parent Question 25 (Do you have another parent or guardian in the 1.28 1.86 1.52 
household same household as you?) 

1 = From single-parent/guardian household 

0 = Not from single-parent/guardian household 


Academic Achievement 


Grade point Question 13 (Indicate your cumulative grade point average 6.60 7.21 6.85 
average (GPA) (GPA) for all classes you took in high school through the end of 
your junior year (last year).) 


Unweighted GPA ona scale of 0.0 to 4.0 converted from letter- 
grade survey responses¢ 


Advanced Question 10 (Have you taken any Advanced Placement (AP) or 2.27 2.97 2.56 
Placement, International Baccalaureate (IB) courses? Include any courses 

International you are taking now.) 

Baccalaureate 


1 = Student has taken an AP or IB course (or both) 
0 = Student has not taken an AP or IB course, or does not know 


College Intentions and Expectations 


course taking 


Intentiontoenroll Question 3 (Which college are you most likely to attend next 24.804 23.08 24.09 
in a two-year year, during the [2016-17 for cohort 1 or 2017-18 for cohort 2] 
college> school year?) 


1 = College name provided is a two-year institution 
0 = College name provided is not a two-year institution 


Educational Question 7 (As things stand now, how far in school do you 5.71 5.49 5.62 
expectations think you will actually get?) 
1 = Student thinks she/he will obtain a bachelor’s degree or 
higher 
0 = Student thinks she/he will obtain less than a bachelor’s 
degree 
College Going Steps Completed Prior to Lottery 
Completed or plans Question 4 (Have you/your family completed and submitted a 0.92 1.36 1.10 


to complete FAFSA FAFSA to apply for financial aid for your education?) 


1 = Yes / No, not yet, but I’m planning to submit a FAFSA 

0 = No, I’m not planning to submit a FAFSA / No, I haven’t 
thought about this yet / No, I don’t know if I or anyone in my 
family will apply / I don’t know what a FAFSA is 


Provided college Question 3 (Which college are you most likely to attend next 0.00 0.00 0.00 
name year, during the [2016-17 for cohort 1 or 2017-18 for cohort 2] 
school year?) 


1 = Provided at least one college name 
0 = Else 


Paid deposit Question 5 (Have you paid a deposit or received a fee waiver to 0.67 1.11 0.85 
enroll at your intended college/university?) 
1=Yes 
0 = No /I don’t know 


a All data are from the student survey, administered in spring 2016 (cohort 1) and spring 2017 (cohort 2). The survey can be found at 
https: //www.reginfo.gov/public/do/PRAViewIC?ref nbr=201503-1850-002&icID=215771. 


b Measure was included as a covariate in impact analysis, regardless of p-value. All other variables were included if p-value is less than .20. 

¢For conversion chart see http://www.collegeboard.com/html/academicTracker-howtoconvert.html. 

4 For this measure to be non-missing, students had to provide a valid college name that they planned to attend and the name had to match a 
college in the IPEDS data. Most students who were missing this measure did not provide a college name (Treatment: 680 students, 24.12%; 
Control: 434 students, 21.88%). Fewer provided an invalid college name or a name that could not be matched to IPEDS because the college was 
outside of the United States or was a small vocational school that was not included in IPEDs (Treatment: 19 students, 0.67%; Control: 24 students, 
1.21%). 

Note: Treatment group (“messaged”) students received the regular GEAR UP supports with the addition of college transition messages; control 
group (“non-messaged” students) received regular GEAR UP supports only. 


Data for all school and grantee characteristics come from multiple administrative sources. Exhibits B.11 and B.12 
show the data source, definition, and percentage of missing data for the school and grantee characteristics, 
respectively. 
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Exhibit B.11 School Characteristics 


Data Source Schools Missing Data? 
Characteristic and Timing Definition/Coding (%) 
Gender CCD, 2015-16 Number of female students in the school divided by the 0.00 
composition total number of students in the school 


Number of male students in the school divided by the total 
number of students in the school 


Racial/ethnic CCD, 2015-16 Number of Hispanic students in the school divided by the 0.00 
composition total number of students in the school 


Number of White, non-Hispanic students in the school 
divided by the total number of students in the school 

Number of Black, non-Hispanic students in the school 
divided by the total number of students in the school 

Number of other race students in the school divided by the 
total number of students in the school 


Historical college § APR, School Number of seniors in the school who immediately enrolled 57.69 
enrollment rate> Year 2014-15 in college divided by the total number of seniors in the 
school 
Title I status CCD, 2015-16 1 = School is eligible for Title I funds 0.00 
0 = School is not eligible for Title I funds 
Rural locale CCD, 2015-16 1 = School is located in a rural setting 0.00 
0 = School is located in a non-rural setting (city, suburb, 
town) 
Completion rate Federal Number of students in the school who completed a FAFSA 0.00 
of the FAFSA Student Aid application divided by the number of seniors in the 
Data Center, school 
2016-17 Cycle 
CCD, 2015-16 


a Missing data rates are provided for study schools. Because the lottery was conducted within schools, all schools have both treatment group 
students and control group students. 

b Missing data rates are high for this characteristic because data could not be obtained from four of the 16 grantees. In addition, data could not be 
obtained from four high schools within the remaining 12 grantees. In total, data were missing for 46 of the 81 high schools. 

Note: APR is Annual Performance Report; CCD is Common Core of Data; FAFSA is Free Application for Federal Student Aid. 
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Exhibit B.12 


Grantee Characteristics 


Characteristic 


Gender 
composition 


Racial 
composition 


Ethnic 
composition 


Historical college 
enrollment rate> 


AP course-taking 
rate 


IB course-taking 
rate 


Title I status 


Rural locale 


Data Source 
and Timing 


APR, FY 2016 


APR, FY 2016 


APR, FY 2016 


APR, School 
Year 2014-15 


APR, FY 2016 


APR, FY 2016 


CCD, 2014-15 
and 2015-16 


CCD, 2014-15 
and 2015-16 


Definition/Coding 


Number of female students served by the grantee divided 
by total number of students served by the grantee 


Number of male students served by the grantee divided by 
total number of students served by the grantee 


Number of White students served by the grantee divided 
by total number of students served by the grantee 


Number of Black students served by the grantee divided by 
total number of students served by the grantee 


Number of other race students served by the grantee 
divided by total number of students served by the 
grantee 


Number of Hispanic students served by the grantee 
divided by total number of students served by the 
grantee 


Number of high school seniors served by the grantee 
immediately enrolling in college divided by total number 
of high school seniors served by the grantee 


Number of students served by the grantee taking AP 
courses divided by total number of students served by 
the grantee 


Number of students served by the grantee taking IB 
courses divided by total number of students served by 
the grantee 


Number of schools served by the grantee that are eligible 
for Title I funds divided by total number of schools 
served by the grantee 


Number of schools served by the grantee that are in a rural 
locale divided by total number of schools served by the 
grantee 


Baccalaureate. 


29 


Grantees Missing Data? 


(%) 
0.00 


18.75 


18.75 


25.00 


6.25 


6.25 


0.00 


0.00 


SE 


a Missing data rates are provided for study grantees. Because the lottery was conducted within schools, all grantees have both treatment group 
students and control group students. 
b Missing data rates are high for this characteristic because enrollment rate data could not be obtained from four of the 16 study grantees. 
Note: AP is Advanced Placement; APR is Annual Performance Report; CCD is Common Core of Data; FY is Fiscal Year; IB is International 


Several of the baseline characteristics described above were also used to examine the effectiveness of college 
transition messaging for particular groups of students or schools (“subgroups”). The study included five subgroups 
that were of policy interest and for which there was existing research suggesting that text-message-based advising 
might have different impacts for students in these groups (Exhibit B.13). 


Exhibit B.13 Rationale for Exploring Effects for Subgroups of Students and Schools 
Subgroup Rationale 


Intention to enroll in a two-year Students at two-year, but not four-year, colleges benefit from text messages (Castleman 

college and Page 2016). Students at two-year colleges often have less access to campus-based 
advisors (Carlstrom and Miller 2013; Gallagher 2010; Scott-Clayton 2015), suggesting that 
text messaging could fill a void in supports for students who intend to enroll in two-year 
colleges. Examining impacts by whether or not a student plans to enroll in a two-year 
college could provide useful information about where to focus resources when engaging 
in college transition text messaging. 


Potential first-generation college | The impacts of text messaging on college enrollment might be larger for students from 

student families where neither parent has attended college than for students from families where 
at least one parent attended college (Bird et al. 2019). Examining impacts by first- 
generation status could help policymakers understand whether text messaging can boost 
the college enrollment rate for first-generation students. 


Gender College enrollment rates are consistently lower for male students than for female students 
(McFarland et al. 2018; Bailey and Dynarski 2011). Further, text messaging could have 
different impacts on male students than on female students (Castleman and Page 2016). 
Examining the impacts by gender could help policymakers understand whether text 
messaging holds promise as means to increase male students’ college enrollment rate. 


Rural locale Proximity to nearby colleges is associated with college enrollment (Turley 2009), with 
students in rural locations less likely to enroll in college (Hu 2003). Examining impacts by 
rural locale could provide useful information about where to focus resources when 
engaging in college transition text messaging. 


FAFSA completion rate of high Text messaging increases college enrollment for students with less access to college 

school advising support in high school and who were not far along in their college planning by 
the end of high school (Castleman and Page 2015). The FAFSA completion rate of a 
student’s high school serves as an indirect measure of access to college supports in high 
school and likelihood of completing important steps on the path to college by the end of 
high school. Examining impacts by high schools above and below the national average 
FAFSA completion rate of 60 percent could provide district, state, and federal 
policymakers with information about where to most effectively target resources devoted 
to text messaging. 


Implementation Measures of College Transition Messaging 


The study collected and analyzed information about implementation of the college transition messaging to measure 
the extent to which advisors and students engaged with the text messages. If the messages did not go out as 
intended or if advisors and students did not engage with the messages, then students’ college going might not be 
affected as expected. The program monitoring system captured data on how the college transition messaging went; 
the implementation measures created from that data appear in Exhibit B.14. There are no missing data on any of 
these measures. 


Exhibit B.14 
| 


Sent all programmed 
messages@ 


Sent each programmed 
message 


Average number of 
programmed messages 
sent 


Sent at least one message 


Not sent messages 
because student opted 
out 


Not sent messages 
because student had an 
invalid phone number 


Not sent messages 
because student is not 
going to college 


Not sent messages 
because student is 
starting college in spring 


Not sent message 
because advisor stopped 
a message 


Responded to at least one 
programmed message 


Average number of 
student responses 


Student sent additional 
non-programmed 
message from advisor 


Measures of Implementation of College Transition Messaging (Treatment Group Only) 


Number (percentage) of students who were sent all college transition text messages 
1 = Student sent all text messages 
0 = Student not sent all text messages 


For each college transition text message, number (percentage) of students who were sent the 
message 


1 = Student sent the text message 
0 = Student not sent the text message 
Average number of college transition text messages sent to students 


Number (percentage) of students who were sent at least one college transition text message 

1 = Student sent at least one GEAR UP text message 

0 = Student not sent at least one GEAR UP text message 

Number (percentage) of students who opted out of the college transition text messages 

1 = Student opted out of the text messages 

0 = Student did not opt out of the text messages 

Number (percentage) of students who had an invalid phone number but did not opt out of the 
text messages 

1 = Student had an invalid phone number but did not opt out 

0 = Student had a valid phone number 


Number (percentage) of students who indicated they were not going to college but did not opt 
out or have an invalid number 


1 = Student indicated s/he not going to college 

0 = Student did not indicate s/he was not going to college 

Number (percentage) of students who said they planned to start college in the spring semester 
but did not opt out, have an invalid number, or indicate that they were not going to college 

1 = Student was a spring starter 

0 = Student was not spring starter 


Number (percentage) of students who were not sent the message because the advisor stopped a 
message 

1 = Student’s advisor stopped the message 

0 = Student’s advisor did not stop the message 


Number (percentage) of students who responded to at least one of the college transition text 
messages, excluding responses that were opt-out requests, invalid number notifications, and 
responses to advisor outreach 


1 = Student responded to at least one college transition text message 
0 = Student did not respond to any college transition text message 


Average number of college transition messages students responded to, excluding responses that 
were opt-out requests, invalid number notifications, or responses to advisor outreach 


Number (percentage) of students to whom an advisor sent at least one additional non- 
programmed text message (excludes programmed messages, auto replies, confirmations, and 
away messages) 


1 = Student to whom advisor sent at least one message in addition to programmed messages 
0 = Student to whom advisor did not send any message beyond programmed messages 


ling 


Average number of Average number of additional non-programmed text messages advisor sent to students, 
additional non- excluding programmed messages, auto replies, confirmations, and away messages 
programmed messages 
sent to students 
Student received advisor | Number (percentage) of students who received at least one message from an advisor in response 
response to the student’s message 

1 = Student sent at least one response from advisor 


0 = Student not sent at least one response from advisor 


Grantee had at least one Number (percentage) of grantees with at least one advisor in attendance at all four training 
advisor attend all webinars 
webinars 1 = Had at least one advisor attend each of the four webinars 


0 = Did not have at least one advisor attend each of the four webinars 


a“All programmed messages” refers to the 37 core messages (35 for cohort 1) and excludes texting logistics messages (such as the initial message 
introducing the messaging) and those sent only to students who were starting college in the spring term. 


Note: Treatment group (“messaged”) students received the regular GEAR UP supports with the addition of college transition messages; control 
group (“non-messaged”) students received regular GEAR UP supports only. 


B.3.2 Estimating the Effects of College Transition Messaging 


Sample Used for Analysis of the Effectiveness of College Transition Messaging 


The study defined the sample as all students who participated in the lottery. This definition helped to ensure that 
the estimates reflected unbiased effects of the college transition messaging on students. Exhibit B.15 diagrams the 
flow of study participants from the recruitment of GEAR UP grantees to the analytic study samples used to 
investigate the effectiveness of college transition messaging on college going measures. 
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Exhibit B.15 Flow from Recruitment of Grantees and Schools to Student Study Sample 


Recruited grantees from FY 2011 grant cycle with GEAR UP 
rising high school seniors in 2015-16 or 2016-17 
(Grantee n=16) 
(School n=81) 
(Student n=11,805 surveyed) 


Excluded the following students from schools who volunteered 
for the study: 
. Did not complete the survey (n=2,128) 

¢ Did not complete the survey before the lottery (n=11) 

¢  Notcollege intending or intention unknown (n=1,121) 
Students determined to be eligible for the lottery e College intending and did not opt into the study (n=3,742) 
(Student n=4,803 ) 


Assigned to receive college transition messages Assigned to not receive college transition messages 
Treatment group Control group 
(Student n=2,819) (Student n=1,984) 


Obtained outcome data from: 
Federal Student Aid office National Student Clearinghouse 
(Student n=1,984) (Student n=1,984) 


Obtained outcome data from: 
Federal Student Aid office National Student Clearinghouse 
(Student n=2,819) (Student n=2,819) 


Created student samples for analysis 
Immediate college enrollment Any college enrollment 
(Student n=4,803) (Student n=4,803) 


Continuous college enrollment College enrollment into second fall 
(Student n=4,803) (Student n=4,803) 
FAFSA submitted for second year FAFSA completed for second year 
(Student n=4,803) (Student n=4,803) 
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Estimation Models 


The study estimated the overall effects of the college transition messaging using a statistical model. Specifically, a 
linear model with a fixed treatment effect was used, which estimated the average treatment effect across all 
participating schools and both cohorts.15 Linear models with conventional standard errors instead of non-linear 
models were used even though the outcomes, such as enrolled versus not enrolled in college, are binary. This is 
because linear models are simpler to estimate and to interpret; they yield unbiased estimates of the treatment 
effect; they yield standard error estimates that are approximately correct even when the underlying data-generating 
process is nonlinear (Judkins and Porter 2016); and they have been used by many random assignment evaluations 
in education. 1© 


Main estimation model 


The following regression model estimated the effects of the study’s college transition messaging: 
(1) Yj = oS Bo; Block; + B,Tij + Dk=1 Ax; BaselineStudentChar ;; + €; 


where ij indexes students and j indexes randomization blocks (students were randomized in blocks defined by school 
and cohort). ¥;; is the value of the outcome, such as on-time college enrollment, for the i“ student in the j” 
randomization block; Block; is 1 if student i was enrolled in block j (for example, school 1, cohort 2), and 0 
otherwise;!” Tyis 1 if student i is randomized to the treatment group in block j and 0 if assigned to the control group 
in block j; BaselineStudentChar ;; are baseline student characteristics, such as academic achievement measures, 
educational expectations, and demographic characteristics, and €;; is random error, assumed to be identically and 
independently distributed. The effect of the transition messaging is given by the parameter £,, which represents the 
difference in the proportion of treatment and control group students achieving the outcome of interest. To test for 
impacts, the study conducted two-tailed t-tests using an alpha-level criterion of p < .05. 


The mean outcomes for the control and the treatment groups were calculated as: 


Y. = the (unadjusted) control group mean of the outcome variable (multiplied by 100 to convert it into a 
percentage) 


Y, = Y, + B, = the (model-adjusted) treatment group mean of the outcome variable (multiplied by 100 to convert 
it into a percentage) 


Covariates. All models controlled for random assignment block fixed effects to improve the precision of the 
estimated effects and to preserve the integrity of the random assignment because of the differing probabilities of 
random assignment to treatment or control group within each block. In addition, the models controlled for baseline 
student-level covariates. (Exhibit B.10 above shows the definition for each student characteristic included in the 
estimation model.) 


Treatment of missing data. As noted previously, there were no missing data for any of the study college going 
measures, and the study used the full randomized sample to analyze the impact of college transition messaging on 


15 This model estimates the overall treatment effect by calculating a weighted average of the treatment effect in each randomization block; that 
is, in each school or cohort within a school if the school participated in the study in both cohorts. 


16 Examples include the evaluations of the Teacher Incentive Fund (Max et al. 2014) and Transfer Incentives for High-Performing Teachers 
(Glazerman et al. 2013). 


17 The inclusion of this set of dummy variables for randomization blocks accounts for the varying ratios of treatment to control students 
(ratios were either 1:1 or 2:1 within a block) by ensuring that a treatment-control difference is calculated within each randomization block 
and that the overall estimate is calculated as a precision weighted mean of within-block treatment-control differences. 
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all college going measures. This implies that no bias was introduced into the estimate of the effect of the study’s 
transition messaging because of missing outcomes. 


The analysis included students with missing covariate values because some students were missing data for 
individual items from the student survey. In these cases, missing values were imputed using the dummy variable 
method. The missing covariate values were replaced with a placeholder (0) and created an indicator for the 
covariate having a missing value, which was included in the model. Simulations by Puma et al. (2009) showed that 
this approach to handling missing covariate data is likely to keep estimation bias at less than 0.05 standard 
deviations. 


Estimation model for subgroups 


In addition to examining the average effects of the college transition messaging on student outcomes, the study also 
investigated the effects of the college transition messaging for subgroups defined by student and school 
characteristics. Exhibit B.13 summarizes the subgroup characteristics that were analyzed. (Exhibit B.10 above 
described how the students were grouped and Exhibit B.11 described how schools were grouped.) These subgroups 
could be defined for both treatment and control students based on their characteristics at the start of the study, 
before the lottery. Therefore, the study can calculate rigorous estimates of the effects of the college transition 
messaging separately for these subgroups. However, because the study is designed to detect effects for the full 
sample, not differences in effects between subgroups, a difference in effects between subgroups can only be detected 
when the true difference is large. 


Models both (a) estimated the impact of the college transition messaging for each of the subgroups and (b) tested for 
differences in impacts among levels of a subgroup indicator. Both types of results are reported; for example, impact 
estimates are reported for both male and female students, and the result is reported for a test of whether the 
magnitude of the impact for male students differs from the magnitude of the impact for female students. Because 
these tests are exploratory, multiple comparison adjustments were not made. It is important to note that with this 
approach, even if there were no significant differences, one might expect to detect at least four differences across the 
70 tests conducted (7 outcomes x 10 subgroup categories) as significant due to chance. 


Student subgroups. To address questions about impacts within subgroups and variation in impacts for subgroups 
of students, an interaction between the subgroup variable and the treatment indicator was added to Equation 1, as 
shown in Equation 2: 


(2) = Wi Boj Block; + BTi; + B2Subgrpi; + B3T;; * Subgrpij + Uk=1 Anj BaselineStudentChar jj + &; 


where f; is the difference in the treatment impact between students in the two subgroups, such as students who are 
potential first-generation college students and students who are not. 


School subgroups. To address questions about impacts for subgroups of schools, a term for the interaction between 
the treatment indicator and the subgroup indicator, such as rural versus non-rural locale. Notably, this model does 
not include a main effect term for the subgroup indicator because it would be co-linear with the block dummies. But 
the interaction term estimates the difference in the average impact between the two subgroups, and impact 
estimates for the two subgroups can be obtained as functions of the coefficients for the treatment indicator and the 
treatment-by-subgroup interaction term. The model has the form 


(3) y= a Boj Block; + BTi; + BoTij * SchSubgrp; + Dk=1Axj BaselineStudentChar j; + &; 


where SchSubgrp, is the school subgroup, such as 1=rural schools and 0=non-rural schools; f, is the treatment 


impact for the reference category, such as non-rural schools; £2 is the difference in the treatment impact between 
the two subgroups; and f, + f, is the treatment impact for the subgroup. 
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Sensitivity Analyses 


The study conducted sensitivity analyses to examine whether the particular statistical models used to estimate the 
effects of college transition messaging affected the findings. The sensitivity analyses tested whether findings 
remained the same when the following three alternative statistical models were used: 


e Logistic regression models were estimated because logistic, rather than linear, regression is commonly used 
for binary outcomes. 


e Aheteroscedasticity adjustment was included to further test the sensitivity of using linear models to 
estimate impacts for binary outcomes. 


e Linear models were re-estimated without covariate adjustment because randomization should yield 
treatment and control groups that are equivalent on both observed and unobserved characteristics, making 
covariate adjustment unnecessary.!8 


Findings from the sensitivity analyses were similar to findings from the main analyses (see Appendix C for details). 


B.3.3. Power Analyses 


Power analyses determine how large a sample is needed for a study with a particular outcome in mind; or, once a 
sample has been identified, to determine how large of an effect can be measured with confidence. This study design 
detects a minimum effect of 3.6 to 4.1 percentage points (depending on the control group’s enrollment rate) for 
immediate college enrollment, based on a targeted number of 4,803 students with an 80 percent probability of 
detecting a statistically significant effect at the 5 percent level. The study-achieved minimum detectable effects 
(MDEs) for immediate college enrollment was 4.6 percentage points. 


The left column of Exhibit B.16 shows the study’s design phase assumptions and expected MDE, whereas the right 
column shows the observed statistics and achieved MDE that the study, as conducted, had 80 percent power to 
detect on immediate college enrollment. The initial power analysis was based on specific assumptions about the 
number of schools recruited, average number of students per school in the study sample, proportion of variance 
explained by covariates, and the success rate in the control group. The achieved MDEs were slightly higher than the 
range of the expected MDEs for two reasons: (1) the student-level R-squared was higher than assumed (19.49 
percent compared to 0.0 percent); and (2) enrollment rates in the control group were close to the high end of the 
range assumed in the study’s design phase. The achieved MDEs for the other study measures—enrollment, 
persistence, and FAFSA renewal—ranged from 4.4 to 4.9 percentage points, shown in Exhibit B.17, all slightly higher 
than the range of the expected MDEs for same two reasons described above. This implies that the study sample was 
powered to detect slightly larger effects than the design phase assumptions. 


Exhibit B.16 Comparison of Design Assumptions and Sample Statistics 


Observed Statistics in 
Design Phase Study Sample for 
Assumption College Enrollment 
Number of schools-by-cohort randomization blocks 86 86 
Average number of students per school 59.30 55.80 
Total number of students 4,803 4,803 
Proportion assigned to treatment group 0.50 0.59 
Student-level R-squared 0.00 0.19 
Minimum Detectable Effect (in SD) 0.082 0.097 


18 The main analyses adjusted for baseline student characteristics in an effort to improve precision of the impact estimates, even though 
covariate adjustment was not necessary to account for baseline differences between treatment and control groups. 
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Success rate in control group 0.50, 0.60, 0.75 


Minimum Detectable Difference (in percentage points) 4.1, 4.0, 3.6 


0.66 
4.6 


Note: SD is standard deviation. 


Exhibit B.17 | Achieved Minimum Detectable Effects (MDEs) for All Outcomes 


Immediate college enrollment after high school 

Any college enrollment during year after high school 

Continuous college enrollment during year after high school graduation 
Continuous college enrollment into the second fall after high school graduation 


FAFSA completion by October 1 of second year after high school graduation 


FAFSA completion by October 1 of first year after high school graduation 


4.6 
4.7 
4.5 
4.4 
4.9 
4.9 
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APPENDIX C. SUPPLEMENTAL TABLES AND INFORMATION ON STUDY FINDINGS 


This appendix includes statistical details on findings presented in the report body, as well as additional findings that 
are not in the report. The details are intended to supplement the findings presented in the report and provide the 
statistical information for readers interested in the technical details of the study’s findings. A summary of the 
findings from studies of similar text messaging programs is also included in this appendix to provide context for the 
study findings in the larger literature. 


C.1 Additional Details about Findings in the Report 


The underlying statistics used to generate the exhibits on the effects of the college transition messaging on college 
going outcomes found in the report may be of interest to some readers. This information is reported for the outcome 
measures discussed in the report (“main” measures) as well as for the exploratory measures listed in Appendix 
Exhibit B.9. Additionally, the exhibits below provide the results for particular groups of students or schools 
(“subgroups”) as well as results from analyses examining the sensitivity of these effects to the statistical model 
chosen. 


Specifically, this section reports on the statistics for the following outcome measures: 
e Immediate college enrollment after high school (main measure) 
e Any college enrollment during year after high school (exploratory measure) 
e Continuous college enrollment during year after high school (main measure) 
e Continuous college enrollment into the second fall after high school (main measure) 
e College enrollment in the second fall after high school (exploratory measure) 
e FAFSA completion for the first fall after high school (exploratory measure) 


e FAFSA completion for the second fall after high school (main measure) 


In the report, Exhibit 4 shows that the college transition messaging had no effect on immediate college enrollment 
after high school. Here Exhibit C.1 presents the estimated effects of the college transition messaging on immediate 
college enrollment after high school and corresponding p-values. It also shows that the college transition messaging 
had no effect on immediate college enrollment after high school for any subgroups of students, but did have 
different effects for one subgroup of schools: Students in rural schools who received the college transition messages 
were more likely to enroll in college immediately after high school than were rural students who did not receive the 
messages.!9 Exhibit C.2 shows that the college transition messaging had no effect on immediate college enrollment 
after high school regardless of the statistical model used for analysis. 


19 Rural schools were less likely to volunteer for the study, which could indicate some selection bias in what types of rural schools participated 
in the study. However, the positive effect for students from rural schools does not seem conclusive. The corresponding effect for students 
from urban schools is negative and there is no reason to believe that the college transition messaging negatively affected these students. 
Further, given the number of statistical tests conducted, at least one significant result is expected to be found by chance alone. 
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Exhibit C.1 Effects on Immediate College Enrollment, Overall and for Subgroups 


Overall Impact 


66.3 66.3 0.0 1.2 992 

Gender 

Male 60.1 58.1 19 2.0 324 

Female 71.0 72.9 -1.9 1.6 246 
F-test of difference p=.133 

Potential First-Generation College Student Status 

Nota potential first-generation 71.5 72.2 -0.7 24 .780 

student 

Potential first-generation student 65.8 65.3 0.5 1.5 761 
F-test of difference p=.690 

Intending to Enroll in a Two-Year College 

Intend to enroll in a four-year college 77.4 76.7 0.7 1.9 .687 

Intend to enroll in a two-year college 52.9 51.8 thal 2.3 .618 
F-test of difference p=.894 

Rural Locale 

Not rural 65.8 68.4 -2.6 1.5 .093 

Rural 67.8 62.9 4.9 21 .020* 
F-test of difference> p=.004 

Completion Rate of the Free Application for Federal Student Aid (FAFSA) 

FAFSA completion rate < 60% 60.0 61.8 -1.8 2.3 25 

FAFSA completion rate > 60% 69.3 68.5 0.8 1.5 589 
F-test of difference p=.336 


a p-Values shown in this column are for tests of whether there was a statistically significant impact for the subgroup category in the row. 
» p-Values shown in this row are for a test of whether impacts statistically differed between the categories of the subgroup in the rows above. 
Notes: Treatment group percentages and impacts are estimated using the study’s regression model. Treatment group (“messaged”) students 
received the regular GEAR UP supports with the addition of college transition messages; control group (“non-messaged”) students received 
regular GEAR UP supports only. 
* Effect is statistically significant at the .05 level, two-tailed test. 
Sample Sizes: 

Overall impact: Treatment group=2,819 students, Control group=1,984 students 

Gender: Treatment group=2,781 students, Control group=1,945 students 

Potential first-generation college student status: Treatment group=2,613 students, Control group=1,840 students 

Intent to enroll in two-year college: Treatment group=2,120 students, Control group=1,526 students 

Locale: Treatment group=2,819 students, Control group=1,984 students 

FAFSA completion rate: Treatment group=2,819 students, Control group=1,984 students 
Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 


Exhibit C.2 


Sensitivity Analyses for Effects on Immediate College Enrollment 


Linear regression (main 66.3 
model) 

Logistic regression 66.2 
Heteroscedasticity adjustment 66.3 
No covariates used in model 65.5 


66.3 


66.3 
66.3 
66.3 


0.0 


-0.1 
0.0 
-0.8 


992 


925 
992 
546 


Notes: Treatment group percentage and impact are estimated using the study’s regression model. Treatment group (“messaged” students) 
received the regular GEAR UP supports with the addition of college transition messages; control group (“non-messaged” students) received 


regular GEAR UP supports only. 


None of the effects is statistically significant at the .05 level, two-tailed test. 
Sample = 2,819 treatment group students and 1,984 control group students 
Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 


Because text messages were sent to the students who delayed college enrollment, the study also examined the effect 
of the college transition messaging on any college enrollment in the first year after high school. Exhibit C.3 shows 
that the college transition messaging had no effect on college enrollment at any point during the year after high 
school. It also shows that the college transition messaging had no effect on college enrollment at any point during 
the year after high school for any subgroups of students or schools. Exhibit C.4 shows that the college transition 
messaging had no effect on college enrollment at any point during the year after high school regardless of the 


statistical model used for analysis. 


Exhibit C.3 


Overall Impact 


Gender 
Male 
Female 
F-test of difference 
Potential First-Generation College Student 


Not a potential first-generation 
student 


Potential first-generation student 

F-test of difference 
Intending to Enroll in a Two-Year College 
Intend to enroll in a four-year college 
Intend to enroll in a two-year college 

F-test of difference 


Rural Locale 
Not rural 
Rural 


F-test of difference 


71.9 


66.3 
75.9 


778 


70.8 


82.0 
60.2 


71.6 
73.0 


72.0 


63.4 
79.0 


81.7 
60.0 


73.8 
69.1 


-0.1 


2.9 


0.1 
p=.940 


-2.2 
3.9 
p=.014 


iL 


1.9 
1.6 


1.8 
Dap 


Impact by School Characteristics 


1.5 
2.0 


.928 


Impact by Student Characteristics 


119 
052 


.997 


.716 


852 
956 


132 
054 


Effects on Any College Enrollment during Year after High School, Overall and for Subgroups 
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Treatment Control 
Group Group 
Students Students Standard 
(%) (%) Impact Error p-Value2 
Completion Rate of the Free Application for Federal Student Aid (FAFSA) 
FAFSA completion rate < 60% 67.5 68.0 -0.5 Ded, 820 
FAFSA completion rate > 60% 74.0 73.9 0.1 1.4 .966 
F-test of difference p=.832 


a p-Values shown in this column are for tests of whether there was a statistically significant impact for the subgroup category in the row. 
b p-Values shown in this row are for a test of whether impacts statistically differed between the categories of the subgroup in the rows above. 
Notes: Treatment group percentages and impacts are estimated using the study’s regression model. Treatment group (“messaged”) students 
received the regular GEAR UP supports with the addition of college transition messages; control group (“non-messaged”) students received 
regular GEAR UP supports only. 
None of the effects is statistically significant at the .05 level, two-tailed test. 
Sample Sizes: 

Overall impact: Treatment group=2,819 students, Control group=1,984 students 

Gender: Treatment group=2,781 students, Control group=1,945 students 

Potential first-generation college student status: Treatment group=2,613 students, Control group=1,840 students 

Intent to enroll in two-year college: Treatment group=2,120 students, Control group=1,526 students 

Locale: Treatment group=2,819 students, Control group=1,984 students 

FAFSA completion rate: Treatment group=2,819 students, Control group=1,984 students 
Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 


Exhibit C.4 Sensitivity Analyses for Effects on Any College Enrollment during Year after High School 


Treatment Control 
Group Students Group Students Standard 
Model (%) (%) Impact Error p-Value 

Linear regression (main 71.9 72.0 -0.1 1.2 928 
model) 

Logistic regression 71.9 72.0 -0.2 0.5 .770 
Heteroscedasticity adjustment 71.9 72.0 -0.1 1.2 929 
No covariates used in model WAZ 72.0 -0.8 1.3 547 


Notes: Treatment group percentage and impact are estimated using the study’s regression model. Treatment group (“messaged”) students 
received the regular GEAR UP supports with the addition of college transition messages; control group (“non-messaged”) students received 
regular GEAR UP supports only. 

None of the effects is statistically significant at the .05 level, two-tailed test. 

Sample = 2,819 treatment group students and 1,984 control group students 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 


In the report, Exhibit 4 also shows that the college transition messaging had no effect on continuous college 
enrollment during the year after high school. Here Exhibit C.5 presents the estimated effects on continuous college 
enrollment during the year after high school and p-values. It also shows that the college transition messaging had no 
effect on continuous college enrollment during the year after high school for any subgroups of students, but did have 
different effects for one subgroup of schools: Students in schools that had a FAFSA completion rate less than or equal 
to 60 percent and who received the college transition messages were less likely to be continuously enrolled in 
college during the year after high school than were students who did not receive the college transition messages.?° 
Exhibit C.6 shows that the college transition messaging had no effect on continuous college enrollment during the 
year after high school regardless of the statistical model used for analysis. 


20‘ The negative effect for students from schools with FAFSA completion rate less than or equal to 60 percent is not conclusive; given the 
number of statistical tests conducted, at least one significant result is expected to be found by chance alone. 
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Exhibit C.5 Effects on Continuous College Enrollment during Year after High School, Overall and for Subgroups 


Overall Impact 


60.2 60.6 -0.4 1.3 764 

Gender 

Male 52.8 50.9 19 2.0 329 

Female 65.6 68.3 -2.7 ily .103 
F-test of difference p=.073 

Potential First-Generation College Student 

Nota potential first-generation 65.7 66.4 -0.7 2.4 .770 

student 

Potential first-generation student 59.7 60.0 -0.2 1.6 888 
F-test of difference p=.865 

Intending to Enroll in a Two-Year College 

Intend to enroll in a four-year college 714 71.5 -0.1 1.9 959 

Intend to enroll in a two-year college 44.7 44.8 -0.1 2.3 973 
F-test of difference p=.995 

Rural Locale 

Not rural 60.6 63.2 -2.7 1.6 088 

Rural 60.1 56.2 4.0 2.2 .066 
F-test of difference> p=.013 

Completion Rate of the Free Application for Federal Student Aid (FAFSA) 

FAFSA completion rate < 60% 52.2 56.8 -4.6 2.3 046* 

FAFSA completion rate > 60% 63.9 62.4 15 1.5 336 
F-test of difference? p=.028 


a p-Values shown in this column are for tests of whether there was a statistically significant impact for the subgroup category in the row. 
» p-Values shown in this row are for a test of whether impacts statistically differed between the categories of the subgroup in the rows above. 
Notes: Treatment group percentages and impacts are estimated using the study’s regression model. Treatment group (“messaged”) students 
received the regular GEAR UP supports with the addition of college transition messages; control group (“non-messaged”) students received 
regular GEAR UP supports only. 
* Effect is statistically significant at the .05 level, two-tailed test. 
Sample Sizes: 

Overall impact: Treatment group=2,819 students, Control group=1,984 students 

Gender: Treatment group=2,781 students, Control group=1,945 students 

Potential first-generation college student status: Treatment group=2,613 students, Control group=1,840 students 

Intent to enroll in two-year college: Treatment group=2,120 students, Control group=1,526 students 

Locale: Treatment group=2,819 students, Control group=1,984 students 

FAFSA completion rate: Treatment group=2,819 students, Control group=1,984 students 
Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 
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Exhibit C.6 Sensitivity Analyses for Effects on Continuous College Enrollment during Year after High School 


Linear regression (main model) 60.2 60.6 -0.4 1.3 .764 
Logistic regression 60.3 60.6 -0.2 0.7 727 
Heteroscedasticity adjustment 60.2 60.6 -0.4 1.3 .766 
No covariates used in model 59.4 60.6 -1.2 1.4 409 


Notes: Treatment group percentage and impact are estimated using the study’s regression model. Treatment group (“messaged”) students 
received the regular GEAR UP supports with the addition of college transition messages; control group (“non-messaged”) students received 
regular GEAR UP supports only. 

None of the effects is statistically significant at the .05 level, two-tailed test. 

Sample = 2,819 treatment group students and 1,984 control group students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 


In the report, Exhibit 5 shows that the college transition messaging had no effect on FAFSA completion in the second 
fall after high school. Here Exhibit C.7 presents the estimated effects on FAFSA completion in the second fall after 
high school and p-values. The college transition messaging had an effect on FAFSA completion in the second fall after 
high school for female students and students in rural schools: Female students who received the college transition 
messages were less likely to complete the FAFSA application in the second fall after high school than were students 
who did not receive the college transition messages. Students in rural schools who received the college transition 
messages were more likely to complete the FAFSA application in the second fall after high school than were students 
who did not receive the college transition messages. Exhibit C.8 shows that the college transition messaging had no 
effect on FAFSA completion in the fall after high school regardless of the statistical model used for analysis. 


Exhibit C.7 Effects on FAFSA Completion in Second Fall after High School, Overall and for Subgroups 


Overall Impact 


54.4 56.7 -2.2 1.4 105 

Gender 

Male 48.7 46.9 1.8 2.1 399 

Female 58.3 63.8 -5.5 1.8 .002* 
F-test of difference p=.009 

Potential First-Generation College Student 

Not a potential first-generation student 60.4 61.5 -1.0 2.6 .690 

Potential first-generation student 53.1 55.6 -2.6 i? 129 
F-test of difference p=.627 

Intending to Enroll in a Two-Year College 

Intend to enroll in a four-year college 64.8 66.8 -2.0 2.0 326 

Intend to enroll in a two-year college 39.8 424 =2.5 2.5 309 
F-test of difference p=.863 

Rural Locale 

Not rural 54.5 58.5 -4.0 1.7 .018* 

Rural 54.8 53.6 1.2 2.3 .619 
F-test of difference p=.073 
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Completion Rate of the Free Application for Federal Student Aid (FAFSA) 


FAFSA completion rate < 60% 48.0 51.6 -3.7 72.05) .140 
FAFSA completion rate > 60% 57.5 59.0 -1.6 1.6 334 
F-test of difference p=.484 


a p-values shown in this column are for tests of whether there was a statistically significant impact for the subgroup category in the row. 
» p-values shown in this row are for a test of whether impacts statistically differed between the categories of the subgroup in the rows above. 
Notes: Treatment group percentages and impacts are estimated using the study’s regression model. Treatment group (“messaged”) students 
received the regular GEAR UP supports with the addition of college transition messages; control group (“non-messaged”) students received 
regular GEAR UP supports only. 
* Effect is statistically significant at the .05 level, two-tailed test. 
Sample Sizes: 

Overall impact: Treatment group=2,819 students, Control group=1,984 students 

Gender: Treatment group=2,781 students, Control group=1,945 students 

Potential first-generation college student status: Treatment group=2,613 students, Control group=1,840 students 

Intent to enroll in two-year college: Treatment group=2,120 students, Control group=1,526 students 

College enrollment steps taken: Treatment group=2,803 students, Control group=1,966 students 

Locale: Treatment group=2,819 students, Control group=1,984 students 

FAFSA completion rate: Treatment group=2,819 students, Control group=1,984 students 
Source: FSA 2017, 2018, 2019; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 


Exhibit C.8 Sensitivity Analyses for Effects on FAFSA Completion in Second Fall after High School 


Linear regression (main 54.4 56.7 =2.2 1.4 105 
model) 

Logistic regression 54.0 56.7 = 2.1) ILS) 081 
Heteroscedasticity 54.4 56.7 =2.2 14 .108 
adjustment 

No covariates used in model 53.8 56.7 -2.8 1.5 054 


Notes: Treatment group percentage and impact are estimated using the study’s regression model. Treatment group (“messaged”) students 
received the regular GEAR UP supports with the addition of college transition messages; control group (“non-messaged”) students received 
regular GEAR UP supports only. 

None of the effects is statistically significant at the .05 level, two-tailed test. 

Sample = 2,819 treatment group students and 1,984 control group students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 


In the report, Exhibit 5 also shows that the college transition messaging had no effect on continuous college 
enrollment into the second fall after high school. Exhibit C.9 presents the estimated effects on continuous college 
enrollment into the second fall after high school and p-values. The college transition messaging had an effect on 
continuous college enrollment into the second fall after high school for male students and students in rural schools: 
Male students and students in rural schools who received the college transition messages were more likely to be 
continuously enrolled in college into the second fall after high school than were students who did not receive the 
college transition messages. Exhibit C.10 shows that the college transition messaging had no effect on continuous 
college enrollment into the second fall year after high school regardless of the statistical model used for analysis. 
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Exhibit C.9 Effects on Continuous College Enrollment into Second Fall after High School, Overall and for Subgroups 


Overall Impact 


50.8 50.8 0.0 1.3 996 

Gender 

Male 44.2 40.1 4.1 2.0 .043* 

Female 55.7 59.0 =82 1.7 059 
F-test of difference p=.006 

Potential First-Generation College Student 

Nota potential first-generation 56.4 55.7 0.7 2.5 .784 

student 

Potential first-generation student 50.5 50.4 0.1 1.6 939 
F-test of difference p=.850 

Intending to Enroll in a Two-Year College 

Intend to enroll in a four-year college 62.8 60.9 19 1.9 312 

Intend to enroll in a two-year college 33.5 34.8 -1.3 2.4 584 
F-test of difference p=.286 

Rural Locale 

Not rural 51.8 54.4 = 29) 1.6 .093 

Rural 49.8 44.8 5.1 2.2 .021* 
F-test of difference p=.004 

Completion Rate of the Free Application for Federal Student Aid (FAFSA) 

FAFSA completion rate < 60% 41.9 45.7 -3.8 Ps) .100 

FAFSA completion rate > 60% 54.9 53.2 1.7 1.5 279 
F-test of difference? p=.049 


a p-Values shown in this column are for tests of whether there was a statistically significant impact for the subgroup category in the row. 
» p-Values shown in this row are for a test of whether impacts statistically differed between the categories of the subgroup in the rows above. 
Notes: Treatment group percentages and impacts are estimated using the study’s regression model. Treatment group (“messaged”) students 
received the regular GEAR UP supports with the addition of college transition messages; control group (“non-messaged”) students received 
regular GEAR UP supports only. 
* Effect is statistically significant at the .05 level, two-tailed test. 
Sample Sizes: 

Overall impact: Treatment group=2,819 students, Control group=1,984 students 

Gender: Treatment group=2,781 students, Control group=1,945 students 

Potential first-generation college student status: Treatment group=2,613 students, Control group=1,840 students 

Intent to enroll in two-year college: Treatment group=2,120 students, Control group=1,526 students 

College enrollment steps taken: Treatment group=2,803 students, Control group=1,966 students 

Locale: Treatment group=2,819 students, Control group=1,984 students 

FAFSA completion rate: Treatment group=2,819 students, Control group=1,984 students 
Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 
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Exhibit C.10 Sensitivity Analyses for Effects on Continuous College Enrollment into Second Fall after High School 


Linear regression (main 50.8 50.8 0.0 1.3 .996 
model) 

Logistic regression 51.2 50.8 0.3 3.5 921 
Heteroscedasticity adjustment 50.8 50.8 0.0 1.3 .996 
No covariates used in model 50.0 50.8 -0.8 15 578 


Notes: Treatment group percentage and impact are estimated using the study’s regression model. Treatment group (“messaged”) students 
received the regular GEAR UP supports with the addition of college transition messages; control group (“non-messaged”) students received 
regular GEAR UP supports only. 

None of the effects is statistically significant at the .05 level, two-tailed test. 

Sample = 2,819 treatment group students and 1,984 control group students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 


Because the study sent text messages to the students who delayed college enrollment or took time off during the 
year immediately after high school, the study also examined the effect of the college transition messaging on college 
enrollment in the second fall after high school. Exhibit C.11 shows the messaging had no effect on college enrollment 
in the second fall after high school; however, students in rural schools who received the college transition messages 
were more likely to be enrolled in college in the second fall after high school than were students who did not receive 
the college transition messages.21 Exhibit C.12 shows that the college transition messaging had no effect on college 
enrollment in the second fall year after high school regardless of the statistical model used for analysis. 


Exhibit C.11 Effects on College Enrollment in Second Fall after High School, Overall and for Subgroups 


Overall Impact 


57.3 57.3 0.0 1.3 .989 
Gender 
Male 50.4 48.0 2.4 2.0 247 
Female 62.2 64.3 =A 1.7 .216 
F-test of difference p=.093 
Potential First-Generation College Student 
Not a potential first-generation 62.5 62.0 0.5 2.5 839 
student 
Potential first-generation student 56.6 56.8 -0.2 1.6 917 
F-test of difference p=.820 
Intending to Enroll in a Two-Year College 
Intend to enroll in a four-year college 68.5 67.3 1.2 1.9 550 
Intend to enroll in a two-year college 42.0 42.7 -0.7 2.4 .764 
F-test of difference p=.540 


21 Rural schools were less likely to volunteer for the study, which could indicate some selection bias in what types of rural schools participated 
in the study. However, the positive effect for students from rural schools does not seem conclusive. The corresponding effect for students 
from urban schools is negative and there is no reason to believe that the college transition messaging negatively affected these students. 
Further, given the number of statistical tests conducted. at least one significant result is expected to be found by chance alone. 
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Impact by School Characteristics 


Rural Locale 

Not rural 58.6 61.0 -25 1.6 122 

Rural 55.7 50.9 4.8 2.2 .031* 
F-test of difference> p=.008 

Completion Rate of the Free Application for Federal Student Aid (FAFSA) 

FAFSA completion rate < 60% 49.7 52.3 =.) 2.4 .283 

FAFSA completion rate > 60% 60.8 59.6 Tot. 1.6 468 
F-test of difference p=.195 


a p-values shown in this column are for tests of whether there was a statistically significant impact for the subgroup category in the row. 
» p-values shown in this row are for a test of whether impacts statistically differed between the categories of the subgroup in the rows above. 
Notes: Treatment group percentages and impacts are estimated using the study’s regression model. Treatment group (“messaged”) students 
received the regular GEAR UP supports with the addition of college transition messages; control group (“non-messaged”) students received 
regular GEAR UP supports only. 
* Effect is statistically significant at the .05 level, two-tailed test. 
Sample Sizes: 

Overall impact: Treatment group=2,819 students, Control group=1,984 students 

Gender: Treatment group=2,781 students, Control group=1,945 students 

Potential first-generation college student status: Treatment group=2,613 students, Control group=1,840 students 

Intent to enroll in two-year college: Treatment group=2,120 students, Control group=1,526 students 

College enrollment steps taken: Treatment group=2,803 students, Control group=1,966 students 

Locale: Treatment group=2,819 students, Control group=1,984 students 

FAFSA completion rate: Treatment group=2,819 students, Control group=1,984 students 
Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 


Exhibit C.12 Sensitivity Analyses for Effects on College Enrollment in Second Fall after High School 


Linear regression (main 


model) 


Logistic regression 
Heteroscedasticity adjustment 


No covariates used in model 


Notes: Treatment group percentage and impact are estimated using the study’s regression model. Treatment group (“messaged”) students 
received the regular GEAR UP supports with the addition of college transition messages; control group (“non-messaged”) students received 
regular GEAR UP supports only. 

None of the effects is statistically significant at the .05 level, two-tailed test. 

Sample = 2,819 treatment group students and 1,984 control group students. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 


C.2 Effects of College Transition Messaging by GEAR UP Grantee 


Researchers have suggested that familiarity with the advisor on the other end of the text message is important to 
students. If the students receiving the messages are disconnected from the advisors sending them, the students 
might not pay attention to the text messages. This study examined whether the effects of the college transition 
messaging differed by GEAR UP grantee to understand whether the effects of the messaging varied by students’ 
familiarity with advisors. This study did not find any differences based on whether the advisors were from the 
students’ high schools or located elsewhere, for example a district or state office or a college. In contrast to the 
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hypotheses discussed but not directly tested in prior studies, the college transition messaging in this study did not 
appear to be any more effective when advisors were familiar staff.22 


This section provides information on exploratory analyses conducted to examine the effects of the college transition 
messaging by GEAR UP grantee. Grantees used advisors who were located in 


e the student’s GEAR UP high school; 
e the student’s college; 
e the GEAR UP central office; or 


e the student’s GEAR UP high school and the student’s college or the GEAR UP central office (combination). 


This section provides graphs that show the effect for each grantee, the overall effect across all grantees, and 
corresponding confidence limits of the effects (Exhibits C.13, C.14, C.15, and C.16). They sort grantees by the location 
of the advisors who sent the text messages to students. The section reports this information for the outcome 
measures discussed in the report (“main” measures). Specifically, information is reported for the following 
outcomes: 


e Immediate college enrollment after high school 
e Continuous college enrollment during year after high school 
e Continuous college enrollment into the second fall after high school 


e FAFSA completion for the second fall after high school. 


Exhibits C.13 - C.16 present the estimated effects of the college transition messaging on each of the four outcomes 
and the corresponding confidence limits for each grantee and overall. Across advisor locations, the college transition 
messaging had no effect on any of the outcomes. The precision of the effects is low for three of the grantees because 
of their small numbers of students who participated in the study (N<50). As observed by the wider confidence 
limits, the conclusions for these grantees should be interpreted with caution. 


22 Avery, Castleman, Hurwitz, Long, and Page 2020; Page, Sacerdote, Goldrick-Rab, and Castleman 2019 
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Exhibit C.13 Sensitivity Analyses for Effect on Immediate College Enrollment after High School, by Grantee 
80 - - 


60 
40 


20 


tet + py tpete- 


Impact estimate 
° 


Number of students 38 85 105 138 184 353 430 801 31 266 46 359 85 536 620 726 4803 


Advisor location School College Central Combination All 
office 


Grantees 


Notes: Effects and confidence limits are estimated using the study’s regression model. “Combination” indicates that advisors were located in the 
student’s GEAR UP high school and the student's college or the GEAR UP central office. 
Sample Sizes: 

Grantee 1: Treatment group=19 students, Control group= 19 students 

Grantee 2: Treatment group=56 students, Control group=29 students 

Grantee 3: Treatment group=68 students, Control group=37 students 

Grantee 4: Treatment group=90 students, Control group=48 students 

Grantee 5: Treatment group=121 students, Control group=63 students 

Grantee 6: Treatment group=235 students, Control group=118 students 

Grantee 7: Treatment group=215 students, Control group=215 students 

Grantee 8: Treatment group=529 students, Control group=272 students 

Grantee 9: Treatment group=20 students, Control group=11 students 

Grantee 10: Treatment group=138 students, Control group=128 students 

Grantee 11: Treatment group=25 students, Control group=21 students 

Grantee 12: Treatment group=179 students, Control group=180 students 

Grantee 13: Treatment group=56 students, Control group=29 students 

Grantee 14: Treatment group=272 students, Control group=264 students 

Grantee 15: Treatment group=314 students, Control group=306 students 

Grantee 16: Treatment group=482 students, Control group=244 students 

All grantees: Treatment group=2,819 students, Control group=1,984 students 
Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 
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Exhibit C.14 Sensitivity Analyses for Effect on Continuous College Enrollment during Year after High School, by 
Grantee 


80 — 


60 
40 


20 


tftp tp tt 


Impact estimate 
° 


Number of students 38 85 105 138 184 353 430 801 31 266 46 359 85 536 620 726 4803 
Advisor location School College Central Combination All 
office 


Grantees 


Notes: Effects and confidence limits are estimated using the study’s regression model. “Combination” indicates that advisors were located in the 
student’s GEAR UP high school and the student’s college or the GEAR UP central office. 

Sample Sizes: Same as Exhibit C.13. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 


Exhibit C.15 Sensitivity Analyses for Effect on Continuous College Enrollment into the Second Fall after High School, 
by Grantee 


Impact estimate 
° 
—_ — 
—_-_ — 
- 
== 
—@— 
- @- 
- 
—- 
-e- 
a 


-20 

-40 

-60 

-80 
Number of students 38 85 105 138 184 353 430 801 31 266 46 359 85 536 620 726 4803 
Advisor location School College Central Combination All 

office 
Grantees 


Notes: Effects and confidence limits are estimated using the study’s regression model. “Combination” indicates that advisors were located in the 
student’s GEAR UP high school and the student’s college or the GEAR UP central office. 

Sample Sizes: Same as Exhibit C.13. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 
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Exhibit C.16 Sensitivity Analyses for Effect on FAFSA Completion for the Second Fall after High School, by Grantee 
80 


60 
40 
20 


. yt ta tt 


-20 


Impact estimate 


-40 
-60 


-80 
Number ofstudents 38 85 105 138 184 353 430 801 31 266 46 359 85 536 620 726 4803 


Advisor location 
School College = Central Combination All 


office 
Grantees 


Notes: Effects and confidence limits are estimated using the study’s regression model. “Combination” indicates that advisors were located in the 
student’s GEAR UP high school and the student’s college or the GEAR UP central office. 

Sample Sizes: Same as Exhibit C.13. 

Source: FSA 2017, 2018, 2019; NSC 2016, 2017, 2018; CCD 2014-15 and 2015-16; IPEDS 2015-16; student survey 2016 and 2017. 


C.3. Findings from Prior Studies of Text Messaging 


A thorough review of prior studies that used text messaging as the primary means of supporting high school 
students to transition into college or of supporting college students to persist to another year of college provides 
context for this study’s findings. Exhibit C.17 summarizes the findings from prior studies that used text messaging as 
the primary form of support for students, though some studies also used other services, such as an initial goal- 
setting activity. This study’s findings of no effect of college transition messaging is consistent with accumulating 
evidence from those prior studies. The exhibit also shows that this study was appropriately powered to detect the 
effects of the text messaging on its college going measures. 
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Exhibit C.17 Summary of Impacts Found in Prior Studies 


5 Impacts 
Study Citation Target Population Design eo Eprolleneey 
Persistence Measure Overall Confidence Interval Subgroup Confidence Intervals@ 

Page, Castleman, and High school seniors RCT Immediate enrollment (-0.2, 6.4) na 
Meyer (2020) 
Bird et al. (2019) First-generation or low- RCT Immediate enrollment (ei 2.5)° na¢ 

income high school 

seniors 
Castleman and Page High school graduates RCT Immediate enrollment (0.0, 5.8)° EFC $0 (1.7, 11.1) 
(2017) EFC not $0 (-4.4, 6.6) 


First-generation (1.0, 12.4) 
Not first-generation (-4.5, 6.5) 


Castleman and Page High school graduates RCT Immediate enrollment (-0.6, 4.4) Dallas (-0.9, 5.7) 
(2015) Boston (-6.7, 3.5) 
Lawrence /Springfield (0.2, 14.0)° 
Avery et al. (2020) High school juniors who RCT Immediate enrollment (-2.6, 0.2) na 
took the PSAT in 15 
states 
High school juniors in 8 RCT Immediate enrollment (-1.2, 4.6) Low-GPA, non-FRL (3.2, 13.8) 
Texas school districts Low-GPA, FRL (-1.9, 3.9) 


High-GPA, non-FRL (-6.7, 1.1) 
High-GPA, FRL (-5.6, 1.8) 


Phillips and Reber Academically on-track RCT Immediate enrollment (-1.9, 3.1)8 Female (-1.4, 5.2) 
(2018) low-socioeconomic- Male (-7.6, 2.6) 
stans Memecnoer Hispanic, Spanish is home language 
juniors (-1.4, 6.4) 
Hispanic, English is home language 
(-7.8, 2.4) 


All other race/ethnicities and 
languages (-5.6, 5.0) 
Parent education less than college 
(1.4, 4.8) 
Parent education at least some college 
(-5.3, 3.3) 


(-1.1, 4.7) Female (-0.9, 6.1) 
Male (-5.7, 5.3) 
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Study Citation 


Target Population 


Design 


Main Enrollment or 
Persistence Measure 


Enrollment in first and 
second fall semesters 


Impacts 


Overall Confidence Interval@ 


Subgroup Confidence Intervals 


Hispanic, Spanish is home language 
(-1.2, 7.4) 

Hispanic, English is home language 
(-6.0, 5.8) 

All other race/ethnicities and 
languages (-4.9, 6.9) 
Parent education less than college 
(-1.5, 5.9) 


Parent education at least some college 
(-3.1, 6.7) 


Page and Gehlbach 
(2017) 


Admitted students 


RCT 


Immediate enrollment 


(20222) 


Committed to Georgia State prior to 
messaging (-0.4, 3.2) 
Did not commit to Georgia State prior 
to messaging (-0.6, 2.2) 


Enrollment at Georgia State 
University 


(-0.6, 3.0) 


Committed to Georgia State prior to 
messaging (0.2, 6.4) 
Did not commit to Georgia State prior 
to messaging (-1.6, 2.8) 


Castleman et al. (2017) 


In-state University of 
Virginia (UVA) 
applicants 


QEDh 


Enrollment at UVA 


(-7.7, 3.3) 


Regular decision (-10.5, 3.7) 
Early action (-9.3, 7.9) 
Below-median SAT score (-12.0, 5.6) 
Above-median SAT score (-8.3, 5.5) 
Underrepresented minority (-7.9, 5.1) 
Non-underrepresented minority 
(-14.1, 7.1) 


Enrollment at highly 
selective college 


(-2.2, 6.0) 


Regular decision (-3.3, 7.7) 
Early action (4.5, 7.7) 
Below-median SAT score (-5.8, 8.4) 
Above-median SAT score (-2.5, 7.7) 
Underrepresented minority 
(-3.3, 6.5) 
Non-underrepresented minority 
(-4.4, 10.6) 
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Main Enrollment or 


Impacts 


Study Citation Target Population Design ; 
Persistence Measure Overall Confidence Interval@ Subgroup Confidence Intervals 
Castleman and Meyer College freshmen QED Fall-to-spring retention in (1.4, 12.0)" EFC $0 (0.3, 8.9) 
(2020) freshman year EFC not $0 (0.6, 14.8) 
Partner college (1.7, 8.7) 
Not partner college (—0.7, 10.3) 
Rural high school (0.1, 15.1) 
Not rural high school (1.5, 8.1) 
Castleman and Page College freshmen RCT Enrollment in sophomore (-4.5, 5.7) Two-year (-0.9, 23.9) 
(2016) fall Four-year (-8.8, 1.8) 
Enrollment in sophomore (-0.5, 9.7) Two-year (1.9, 26.3) 
spring Four-year (-4.4, 6.6) 
Continuous enrollment (-2.4, 9.0) Two-year (1.0, 26.6) 
during sophomore year Four-year (-6.8, 5.4) 
Mabel, Castleman, and College students RCT Re-enrollment in spring (-0.5, 4.5) Predicted dropout risk (-1.6, 1.2) 
Bettinger (2017) after text messaging 
Graduation in year of text (-3.2, 2.6) Predicted dropout risk (—0.9, 1.9) 
messaging 
Graduation in fall of text (-2.9, 1.1) Predicted dropout risk (-1.5, 0.5) 
messaging year 
Graduation in spring of text (-1.9, 3.1) Predicted dropout risk (—0.1, 2.3) 
messaging year 
Oreopoulos, Patterson, College students RCT Enrollment in semester Univ. Toronto main campus (-0.2, na° 
Petronijevic, and Pope after text messaging 0.6)' 
(2019) : 
Univ. Toronto commuter campus 
(-0.1, 0.3) 
Western Governors Univ. (-3.2, 
0.0) 
Dobronyi, Oreopoulos, College students RCT Enrollment in year after (-6.2, 2.8) na‘ 


and Petronijevic (2019) 


text messaging 
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Main Enrollment or Impacts 


Study Citation Target Population Design : 
Persistence Measure Overall Confidence Interval@ Subgroup Confidence Intervals 
Page, Sacerdote, College students RCT Total months enrolled (-8.9, 25.3)i FAFSA not completed (-15.1, 24.5) 
Goldrick-Rab, and FAFSA not completed but was 
Castleman (2019) enrolled in semester prior to text 
messaging (-13.7, 42.4) 
Graduation (-0.6, 4.7) FAFSA not filed (—2.9, 2.9) 


FAFSA not completed but was 
enrolled in semester prior to text 
messaging (-6.3, 3.5) 


EFC is Expected Family Contribution; FAFSA is Free Application for Federal Student Aid; FRL is free or reduced-price lunch eligible; GPA is grade point average; na is not applicable; QED is quasi- 
experimental design; RCT is randomized controlled trial. 

Notes: Bold indicates a statistically significant impact at p<.05. 

a A confidence interval is a measure of the precision of the impact estimate. If a study of text messaging were repeated many times, the estimated impact would be expected to fall within the 
confidence interval 95 percent of the time. A narrow 95 percent confidence interval (for example, spanning 3 or 4 percentage points) reflects a more precise estimate, whereas a wide 95 percent 
confidence interval (for example, spanning 10 percentage points) reflects a less precise estimate. 

" Results shown are for the “advising” treatment arm, which is the most similar to this study because students in the treatment arm were invited to text back to connect to an advisor, whereas 
students in the other treatment arms received information-only outreach. The authors state that they “consistently find no effect [of] these messages on student enrollment” regardless of treatment 
arm (p. 3). 

© Study does not provide subgroup results for this treatment arm. 

“ Results shown are for the treatment arm that included students only, not students and parents. 


‘ Sample sizes were not provided for the other subgroups discussed (that is, subgroups by GPA, FAFSA completion status, and college plans), so confidence intervals could not be calculated. 

£ Study also examined but did not report subgroup results by student race, ethnicity, and gender. 

* All results shown are for the “complete” treatment arm, which included both automated messages and access to a personal virtual advisor and is the most similar to this study. Results are not 
sensitive to the choice of treatment arm. 

h Results reported here are for the authors’ “main” comparison group - students who intended to apply for financial aid but did not opt in to the messages. Results are not sensitive to the choice of 
comparison groups. 

‘Results shown are for the pooled sample that received reminder messages (including students in the goal-setting only treatment arm and the goal-setting plus mindset messages treatment arm). 
Study does not provide results pooled across sites. 

’ Results shown are for the most comprehensive treatment arm, which included information, reminders, and invitations to receive individualized assistance from an advisor via text. Results are not 
sensitive to the choice of treatment arm. 
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APPENDIX D. BAYESIAN ANALYSIS OF IMMEDIATE COLLEGE ENROLLMENT RESULT 


All of the results presented elsewhere in this appendix are based on classical (or traditional), frequentist statistical 
analysis approaches to estimating the effects of the college transition messaging on college going measures.”3 This 
appendix reports effect of college transition messaging on immediate college enrollment after high school using a 
Bayesian approach. Bayesian analysis is an alternative method to reporting p-values to determine the effectiveness 
of an intervention. In recent years, and possibly partially due to the American Statistical Association’s “ASA 
Statement on Statistical Significance and P-Values,” there has been renewed interest in alternatives to reporting p- 
values (Wasserstein and Lazer 2016). Some statisticians and researchers suggest that consumers of research can 
more easily understand the results of Bayesian analyses they can results from traditional, frequentist approaches to 
impact analyses.24 


This appendix describes the Bayesian analysis that this study conducted and presents its results. Because the 
application of Bayesian analysis is relatively new in education research,25 the methods used are described below in 
sufficient detail that other researchers might scrutinize, replicate, and improve on the approach. Specifically, the 
description of the methodology is presented so that an analyst with a moderate amount of experience conducting 
statistical analyses and programming equations with statistical software such as SAS, R, Stata, or SPSS could 
reproduce the Bayesian analysis results, and therefore fully understand the approach, using the data presented 
herein. Section D.6 contains all of the technical details and can be skipped by readers not wishing to delve into them. 


D.1 A Bayesian Approach to Interpreting Study Results in the Context of Other Studies 


Using the traditional, frequentist approach, the current study found practically no evidence that the college 
transition messaging made a difference in whether students enrolled in college immediately after high school. This 
study, however, is only one of several studies examining the effects (or impacts) of text-message-based advising on 
college enrollment after high school. That collection of studies all used similar designs that compared students who 
were offered text messaging versus students who were not. Despite these similarities, some of the studies concluded 
that text messaging had a positive effect on college enrollment immediately after high school, whereas others found 
no evidence of such an impact. 


The conclusions for each of those studies are based on a traditional approach to investigating potential impacts that 
relies solely on the evidence gathered during a single study and on the results of tests of statistical significance. 


23 For each outcome, the classical, frequentist statistical approach produces an estimate of the size of the intervention effect on the outcome 
and a statement, in the form of a p-value, about whether the estimate of the size of the intervention impact was statistically significant. For 
example, in the current study, that approach estimated a 0.01 percentage point increase in immediate college enrollment after high school 
due to the college transition messaging (the intervention), reported a p-value of .99, and concluded that the intervention effect was not 
significantly different than zero. The p-value of .99 from the classical analysis indicates that the result is highly consistent with what would 
have been expected if college transition messaging had no effect on the percentage of students who enroll in college immediately after high 
school. Specifically, the classical analysis estimates that given the sample size, study design, and analytic model, there would have been a 99 
percent chance of obtaining an intervention effect estimate of 0.01 percentage points or larger when, in fact, the true effect was zero. 


24 A stated goal of a 2017 research methods meeting sponsored by the Office of Planning, Research, and Evaluation titled “Bayesian Methods 
for Social Policy Research and Evaluation” was to “encourage attendees to question their assumptions around traditional frequentist 
approaches and explore the leading alternative, Bayesian methods” (OPRE 2017). All four of the papers posted as products resulting from 
that meeting indicate that the primary motivator for considering Bayesian over traditional, frequentist approaches to impact analyses in the 
social sciences is that research consumers will more easily understand the probabilistic statements produced from Bayesian analyses than 
they will results from traditional, frequentist approaches to impact analyses. 


25 For example, in a 2018 review of the use of Bayesian statistics in educational research, authors Konig and van de Schoot identified only one 
study that used results of prior studies in its Bayesian analysis, and that study was not an impact evaluation examining impacts on student 
outcome measures. 
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However, rather than rely on a single study to understand whether text messaging or any other intervention works, 
the Bayesian approach presented here combines evidence from a single study with information from prior studies.?° 


The analyses described in this section interpret the results from the current study in the context of what has been 
learned from prior studies.’ The results of the Bayesian analysis indicate that: 


e =Itis alittle more likely that college transition messaging has a positive effect rather than a negative effect on 
college enrollment immediately after high school, but it is unlikely that the messaging increases or 
decreases the percentage of students who immediately enrollment by more than 1 percentage point. 


e Specifically, there is a 58 percent chance of a positive effect versus a 42 percent chance of a negative effect, 
and a 20 percent chance of a 1 percentage point increase versus a 10 percent chance of a 1 percentage point 
decrease in immediate enrollment due to the messaging. The probabilities that the messaging increases or 
decreases immediate enrollment by 2 percentage points or more are each less than 3 percent. 


There are multiple decision-points in any Bayesian analysis, and different choices can be made at each point. 
Therefore, it is important to determine whether the results found through this study’s chosen approach to Bayesian 
analysis are sensitive to particular decisions. This section first presents a primary approach to the analysis, followed 
by a series of sensitivity analyses. The sensitivity analyses presented include an alternative approach to identifying 
relevant prior studies, alternative assumptions about the effects of publication bias,28 an alternative statistical 
method to estimate the “prior distribution,” an alternative statistical method to estimate the “posterior 
distribution.”29 They also include the use of what Deke and Finucane (2020) refer to as “benchmark priors” — 
estimates of the mean and standard deviation of the “prior distribution” that they have calculated from all studies in 
the What Works Clearinghouse™ (WWC) database that met evidence standards with or without reservations, and 
that they argue represent reasonable, broad, evidence-based priors for use with Bayesian analyses of the effects of 
education interventions. 


Differences across these dimensions in the primary approach and the sensitivity analyses are summarized in Exhibit 
D.1, with the key difference of each approach in red text. 


26 Deke and Finucane 2019; Konig and van de Schoot 2018. 


27, Bayesian approaches do not necessarily have to use results from prior studies. Some Bayesian analyses use “non-informative priors”; that is, 
use prior information based on expert judgement. In the current application of Bayesian analysis, the study team is following the 
recommendations of Deke and Finucane (2019) and using prior information obtained from prior studies. 


28 Publication bias occurs when the results of the study, such as a finding of a large effect or no effect, influence whether the study is published 
or the results are reported. If there is publication bias, small effects or no effects, are less likely to be reported. 


29 The choices of these sensitivity analyses were guided by recommendations in Deke and Finucane (2019, 2020). 
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Exhibit D.1 Details about Key Dimensions of the Primary Approach and Each Sensitivity Analysis 
Analysis Method Analysis Method Used to 
Used to Estimate Estimate 
Analysis Prior Studies Publication Bias Prior Distribution Posterior Distribution 
Relevant prior studies No adjustment for Closed-form equations Closed-form equations 
Primary Approach identified by literature potential publication bias used to estimate mean used to estimate mean and 
review and standard standard deviation 
deviation 
Relevant prior studies No adjustment for Closed-form equations Closed-form equations 
fe 4 identified in What potential publication bias used to estimate mean used to estimate mean and 
Sensitivity Analysis ; re 
1 Works Clearinghouse and standard standard deviation 
database of reviewed deviation 
studies 
Relevant prior studies | Adjustment made for Closed-form equations Closed-form equations 
Pm . identified by literature potential publication bias used to estimate mean used to estimate mean and 
Sensitivity Analysis ; s oss 
3 review — assume published and standard standard deviation 
effects are inflated by a deviation 
factor of two 
Relevant prior studies Adjustment made for Closed-form equations Closed-form equations 
Sensitivity Analysis _ identified by literature potential publication bias used to estimate mean used to estimate mean and 
4 review — assume that all and standard standard deviation 
published effects are zero deviation 
Relevant prior studies No adjustment for Iterative procedure Closed-form equations 
Sensitivity Analysis identified by literature potential publication bias used to estimate mean used to estimate mean and 
4 review and standard standard deviation 
deviation 
Relevant prior studies No adjustment for Closed-form equations Iterative procedure used 
Sensitivity Analysis _identified by literature potential publication bias used to estimate mean to estimate mean and 
5 review and standard standard deviation 


Sensitivity Analysis 
6 


Sensitivity Analysis 
7 


All studies in What 
Works Clearinghouse™ 
database of reviewed 
studies that met 
standards with or 
without reservations 


All studies in What 
Works Clearinghouse™ 
database of reviewed 
studies that met 
standards with or 
without reservations 


No adjustment for 
potential publication bias 


“File drawer correction” 
as reported in Appendix 
D of Herrmann et al. 
(2019) 


deviation 


Mean and standard 
deviation of prior 
distribution reported 
in Table D.1 of 
Herrmann et al. 
(2019) 


Mean and standard 
deviation of prior 
distribution reported 
in Table D.1 of 
Herrmann et al. 
(2019) 


Use of “Benchmark Priors” Recommended by Deke and Finucane (2020) and reported in Herrmann et al. (2019) 


Closed-form equations 
used to estimate mean and 
standard deviation 


Closed-form equations 
used to estimate mean and 
standard deviation 


With two exceptions, the results from the sensitivity analyses point to similar conclusions as the primary analysis— 
that there is a greater than a 50 percent chance that the effect of college transition messaging on college enrollment 
immediately after high school is greater than zero, but it is unlikely that the effect is larger than 1 or 2 percentage 
points. In the two exceptions, described below, the sensitivity analyses resulted in an estimate of equal chances of 
positive and negative effects, but these results agreed with the others in that the probabilities were low that effects 
were greater than 1 or 2 percentage points in either direction. 


The current study found that the college enrollment rates immediately after high school were nearly identical for 
students who were offered the college transition messages compared to those who were not offered college 
transition messages. The enrollment rates were 66.34 percent and 66.33 percent, respectively. The results from the 
traditional analysis described in the main report included a statistical test for this small difference of 0.01 
percentage points that resulted in a p-value equal to .99. That p-value from the traditional analysis indicates that the 
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result is highly consistent with what would have been expected if college transition messaging had no impact on the 
percentage of students who enroll in college immediately after high school. 


Despite the current study’s very small, near zero, estimate of the effect of the messaging on immediate college 
enrollment, it is not surprising that the conclusions from the Bayesian analysis indicate a greater than 50 percent 
chance of a positive, yet small, effect. In the broader context, all but one of the prior studies (identified using the 
primary approach) used in the Bayesian analysis had positive estimates of the effect of text messaging, and all of the 
estimated effects were 3 percentage points or less. 


D.2 Identifying Relevant Prior Studies 


A central aspect of the Bayesian analysis as implemented here is that it incorporates prior knowledge into the 
current analysis. Holzwart and Wright (2018) suggest that determinations about what prior knowledge to 
incorporate should often be made in collaboration between researchers and stakeholders. Deke and Finucane 
(2019) recommend using only prior knowledge that comes from existing studies and not using expert judgement or 
non-informative prior information such as assuming an arbitrary mean as the basis of prior knowledge. Based on 
their simulation results, Deke and Finucane argue that even when prior evidence is not available from interventions 
that are closely matched to the current intervention, that the use of imperfect but thoughtfully selected evidence 
from prior studies is the preferred approach for conducting Bayesian analyses and will produce useful results. For 
education interventions, they specifically recommend the use of a database of reviewed studies as a source of prior 
information on plausible distributions of intervention effects, and they also recommend use of their “benchmark 
priors,” which were calculated from all evidence reported in the WWC database of reviewed studies that met 
standards with or without reservations. Additionally, they recommend assessing the sensitivity of results to the 
choice of which prior studies are used. 


Given the timeline of this study, collaboration with stakeholders was not possible for the current Bayesian analysis. 
The study team did, however, make the decision to restrict the “prior knowledge” that informs the current Bayesian 
analysis to results from prior studies. In addition, in order to assess the sensitivity of the results to the choice of 
prior studies, the team used three different approaches to identifying relevant prior studies. 


The first approach used a broad-based literature review strategy, whereas the second used only studies in the WWC 
database of reviewed studies that met standards with or without reservations. The third approach used “benchmark 
priors.” The first approach was chosen as the primary approach because it identified studies of programs that were 
closely aligned with the text-message-based advising implemented in this study. Specifically, the studies that were 
identified using the first approach each estimated the impact of text messaging on college enrollment immediately 
after high school, whereas the studies identified using the second approach involved programs and strategies 
beyond text messaging that targeted difficulties students face in transitioning to college. The first approach 
identified eight relevant studies, whereas the second identified five. Only two studies were identified by both 
approaches.?° 


D.2.1 Literature Review Approach - Primary Approach 


The primary approach to selecting prior studies was based on an extensive review of the literature for any studies of 
text messaging as the sole or primary program delivered to high school students before they entered college. These 
included programs that primarily used text messaging to send reminders and encouragement to students but might 
also have included other activities, such as an initial online goal-setting exercise. To be included, studies examined 
impacts on immediate college enrollment after high school. Studies also had to use a design that was likely to 


30 Though Deke and Finucane (2019) recommend conducting sensitivity analyses based on “a broader or narrower population of prior 
studies,” the study team was not able to identify any sensible ways of broadening the population of prior studies beyond the literature- 
review-based and the WWC-based approaches to identifying studies. 
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produce rigorous evidence of the effect of the program.3! When multiple variations of text messaging were 
examined—that is, in studies with multiple treatment arms, the impact estimate for the treatment arm closest to the 
college transition messaging examined in this study was used. A summary of the current study and the prior studies 
identified through the literature review approach is in Exhibit D.2. All impacts and standard errors are from the full 
study samples, not for particular groups of students (“subgroups”). 


Exhibit D.2 Summary of Impacts from Current Study and Relevant Prior Studies Identified by Literature Review 
Approach 


Not reviewed Linkow et al. (2020) 4,803 0.0001¢ 0.012 
| Prior Studies 
eee Castleman and Page (2015) 5,753 0.0194 0.013 
Not reviewed Bird et al. (2019) 271,365 0.007¢ 0.009 
Not reviewed Page and Gehlbach (2017) 7,489 0.010 0.006 
Not reviewed Castleman and Page (2017) 3,392 0.0298 0.015 
ie Ce Page, Castleman, and Meyer (2020) 17,731 0.031h 0.017 
Not reviewed Phillips and Reber (2018) 6,640 0.006: 0.013 
Not reviewed Avery et al. (2020) National 70,285 -0.014i 0.006 
Not reviewed Avery et al. (2020) Texas 21,001 0.0173 0.015 


a Impacts are the studies’ estimated effects of text messaging on the proportion of students who enrolled in college immediately after 
high school. Multiplying the proportions by 100 converts them to percentage points. For example, the results from Castleman and 
Page (2015) indicate that the percentage of treatment group students who enrolled in college immediately after high school was 1.9 
percentage points higher than the percentage of control group students who did so. 

» Impact standard errors are measures of the precision of the impact estimates and are needed, in addition to the impact estimates, to 
replicate the Bayesian analysis results. Smaller standard errors indicate greater precision. 

¢ Linkow et al. (2020): Effect of text-message-based advising. 

4 Castleman and Page (2015): Effect of text-message-based advising. 

e Bird et al. (2019): Effect of multi-modal campaign (text message, email, and postal) encouraging students to complete the FAFSA. 
Estimates here focus on treatment arm that included offer of one-on-one virtual advising. 

f Page and Gehlbach (2017): Effect of text-message-based outreach sent by a virtual assistant/artificially intelligent (AI) system. 
Admissions staff responded via the system when the AI could not answer. 

8 Castleman and Page (2017): Effect of text-message-based information and advising to students. Estimates here focus on treatment 
arm that included texts to students only, not parents. 

h Page, Castleman, and Meyer (2020): Effect of text-message-based information and assistance to complete the FAFSA. Students could 
write back for assistance or follow up in-person with their school counselor. 

' Phillips and Reber (2018): Effect of information, reminders, and advising delivered via text messages, the internet, phone, email, and 
social networking platforms. Estimates here focus on treatment arm that included offer of one-on-one virtual advising. 

i Avery et al. (2020): Effect of text-message-based advising. National sample: Text-message-based advising from virtual advisors. 
Texas sample: Text messages from high school counselors with option to follow up via text or in-person. 


31 This means that to be included, the study had to compare measures of immediate college enrollment of treatment students (that is, students 
who were offered the text messaging) to those of comparison students (that is, students who were not offered the text messaging). If 
assignment to treatment or comparison groups was not via lottery, then in order to be included, the study needed to provide evidence that 
the treatment and comparison groups were similar prior to the start of the text messaging. 
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D.2.2 What Works Clearinghouse™ (WWC) Search Approach - Sensitivity Analyses 


The second approach used only studies in the WWC database of reviewed studies.?? Studies were those that met 
WWC standards with or without reservations, examined impacts on immediate college enrollment after high school, 
and offered advising to college-intending seniors targeted to the issue of transitioning to college. This approach 
identified eight studies.33 Exhibit D.3 summarizes the findings from the current study and the prior studies 
identified by the WWC search approach. 


Exhibit D.3 Summary of Impacts from Current Study and Relevant Prior Studies Identified by 
WWC Search Approach 


Current Study 


Not reviewed Linkow et al. (2020) 4,803 0.0001¢ 0.012 
Met without Castleman, Arnold, and Wartman 162 0.134 0.08 
reservations (2012) 

Met without Castleman, Page, and Schooley (2014) 2,166 0.035¢ 0.017 
reservations 

Met without Castleman and Page (2015) 5,753 0.019! 0.013 
reservations 

Met without Castleman, Owen, and Page (2015a) 3,281 0.035 0.019 
reservations 

Met without Castleman, Owen, and Page (2015b) 1,602 0.0018 0.015 
reservations 

Met with reservations Barr and Castleman (2017) 2,422 0.070: 0.016 
Met without Page, Castleman, and Meyer (2020) 17,731 0.03) 0.017 
reservations 

Met with reservations Bettinger and Evans (2019) 122,276 0.000* 0.008 


a Impacts are the studies’ estimated effects of the program on the proportion of students who enrolled in college immediately after 
high school. Multiplying the proportions by 100 converts them to percentage points. For example, the results from Castleman and 
Page (2015) indicate that the percentage of treatment group students who enrolled in college immediately after high school was 1.9 
percentage points higher than the percentage of control group students who did so. 


bImpact standard errors are measures of the precision of the impact estimates and are needed, in addition to the impact estimates, to 
replicate the Bayesian analysis results. Smaller standard errors indicate greater precision. 


¢ Linkow et al. (2020): Effect of text-message-based advising. 


4 Castleman, Arnold, and Wartman (2012): Effect of providing proactive college counseling to low-income students during the 
summer. 


¢ Castleman, Page, and Schooley (2014): Effect of two to three hours of summer counseling. 

f Castleman and Page (2015): Effect of text-message-based advising. 

8 Castleman, Owen, and Page (2015a): Effect Summer monitoring and follow up via virtual platform used by high school counselors. 
4 Castleman, Owen, and Page (2015b): Effect of summer outreach by public high school or university based counselor. 

i Barr and Castleman (2017): Effect of “Bottom Line” advising program. Advisors meet with students for an hour every three or four 
weeks during senior year. 


i Page, Castleman, and Meyer (2018): Effect of text-message-based information and assistance to complete the FAFSA. Students could 
write back for assistance or follow up in-person with their school counselor. 


Bettinger and Evans (2019): Effect of near-peer mentoring by recent college graduates to high school seniors. 


32 The study team downloaded the database on April 24, 2019. 


33 Five of these studies were included in the What Works Clearinghouse™ Summer Counseling Intervention Report 
(https://ies.ed.gov/ncee/wwc/Docs/InterventionReports/wwc summer counseling 032718.pdf). 
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D.2.3 Use of “Benchmark Priors” 


Deke and Finucane (2020) argue that for education interventions, such as the one being evaluated in the current 
study, benchmark priors calculated from all evidence reported in the WWC database of reviewed studies that met 
standards with or without reservations provide a reasonable set of prior evidence. They recommend that 
benchmark priors be included in sensitivity analyses. Therefore, the study included benchmark priors. The 
benchmark numbers were reported in Deke and Finucane (2020) and in Appendix D of Herrmann et al. (2019). 


D.3. Assumptions about and Approaches to Addressing Publication Bias 


Because publications served as the source for prior knowledge in the current Bayesian analysis, it is important to 
have an understanding of what studies may or may not be published and to be explicit about any assumptions made. 
Publication bias occurs when the results of a study influence whether the study is published or the results are 
reported. For example, if null or small effects are less likely to be reported, then there is publication bias. Because 
there is no way to know whether publication bias has occurred, researchers recommend conducting sensitivity 
analyses with several different assumptions regarding its presence (Deke and Finucane 2019). 


D.3.1 Assumptions about Publication Bias 
According to Deke and Finucane (2019), three different assumptions to test are the following: 


1. Assume that the literature can be taken at face value and do no adjustments to the results from the prior 
studies. 


2. Assume that the impacts in the literature are overstated by a factor of two. 


3. Account for possible overestimation of impacts in the literature more stringently by assuming that, on 
average, past impacts in this set of interventions equal zero. 


Because there are multiple published studies where the effect of text messaging on immediate college enrollment 
after high school was tested, the primary approach assumes no publication bias and takes the literature at face 
value. The assumption is that it would be a rare occurrence for a study team to go through the trouble of 
implementing an intervention and conducting a rigorous evaluation of the intervention’s effects on immediate 
college enrollment after high school, only to fail to report the results in the public domain. But, of course, it is 
unknown whether the assumption about the lack of publication bias is correct. Therefore, two sensitivity analyses 
were conducted using the latter two assumptions. 


D.3.2. Approaches to Addressing Publication Bias 


To carry out the primary approach, which takes the literature at face value, the impacts shown in Exhibit D.2 above, 
in the column labeled “Impact on Immediate College Enrollment,” were used in the Bayesian analysis without any 
adjustment to the numbers. 


To carry out the sensitivity analysis under assumption 2, which assumes impacts in the literature are exaggerated by 
a factor of two, six additional simulated studies were added to the analysis. This approach assumed that these 
additional six studies were otherwise identical to the six shown in Exhibit D.2 in their sample sizes and standard 
errors, but that these six additional, unpublished studies had found impacts that were equal to zero. This sensitivity 
analysis therefore includes a total of 12 prior studies. 


Assumption 3 takes the stringent assumption that the average of prior impacts of text messaging interventions was 
zero. The Bayesian analysis uses the mean and standard deviation of impact estimates from prior studies. To 
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operationalize this sensitivity analysis, the mean of the impact estimates from prior studies was set to zero, and used 
the standard deviation of the impact estimates that was calculated using the primary approach.*4 


In addition to the approaches described above, the study also used a set of “benchmark priors” reported in 
Herrmann et al. (2019) that included an adjustment for publication bias in a sensitivity analysis. 


D.4 Analysis Methods Used to Estimate Prior and Posterior Distributions 


The Bayesian analysis involved three broad steps: 


Step 1: Calculate the mean and standard deviation of impact estimates from relevant prior studies. The results 
are called “the priors.” 


Step 2: Use the priors from Step 1 in combination with the results from the current study to estimate the mean 
and standard deviation of impacts in the “posterior distribution.” 


Step 3: Using the mean and standard deviation of the posterior distribution, calculate the probability that the 
intervention impact is greater than a particular number (for example, greater than zero or greater than 2 
percentage points). 


D.4.1 Two Methods to Calculate the Mean and Standard Deviation from Relevant Prior Studies 


The study team identified two approaches for calculating the mean and standard deviation from relevant prior 
studies. One was easier to implement and required no specialized software beyond what any analyst would have at 
their disposal, the other approach required specialized software but could be considered to be more “state-of-the- 
art’”—so the study implemented both. The first approach used closed-form equations that can be easily programmed 
in any statistical analysis package such as SAS, Stata, SPSS, R, or even Excel. The second approach used iterative 
procedures for the calculations implemented with the “RStan” package in R software. (Section D.6, Technical Details, 
describes how these approaches were implemented, for interested readers.) 


The results were not sensitive to choice of either of the two approaches. The closed-form-equations approach was 
designated as primary and the iterative approach as a sensitivity analysis because the former was easier to 
understand and implement. Two studies (Michalopoulos 2012; Herrmann et al. 2019) that have implemented 
Bayesian analyses using priors from relevant prior studies do not provide sufficient detail to ascertain the methods 
used to calculate the mean and standard deviation from the prior studies they used. 


D.4.2 Two Methods to Estimate the Mean and Standard Deviation of the Posterior Distribution 


Similar to above, the study identified two methods for estimating the mean and standard deviation of the posterior 
distribution, and for similar reasons both were implemented. The first approach similarly used closed-form 
equations. The second approach similarly used iterative procedures for the calculations, but implemented them 
using the “GenMod” procedure in SAS software.?5 (Section D.6, Technical Details, describes how these approaches 
were implemented, for interested readers.) 


Again the results were not sensitive to choice of either of the two approaches. The closed-form-equations approach 
was designated as primary and the iterative approach as a sensitivity analysis because the former was easier to 
understand and implement and could be implemented using only summary statistics from the current and prior 
studies, whereas the latter approach required the use of the original student-level micro data. Because the study 


34 An alternative approach to operationalizing the recommendation by Deke and Finucane (2019) to assume all prior studies had a mean of 
zero would have been to specify “flat priors,” wherein the mean is assumed to be zero and the standard errors are assumed to be infinitely 
large. This approach was not utilized because they strongly argue against the use of flat priors. 


35 Michalopoulos (2012) used the closed-form-equations approach. Herrmann et al. (2019) does not provide detail on the method used to 
estimate the mean and standard deviation of the posterior distribution. 
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team no longer had access to the original student-level micro data, nor would analysts wishing to reproduce the 
results from the Bayesian analyses presented herein, the team simulated student-level micro data to assess the 
sensitivity of the results to the choice of the closed-form-equations approach versus the iterative approach. (Details 
on the simulation of data are provided in Section D.6.) 


D.5 Results of Primary and Sensitivity Analyses 


The Bayesian analysis incorporates evidence from prior studies into the interpretation of the impact found in the 
current study. The Bayesian analysis results indicate a slightly higher likelihood that college transition messaging 
advising had a positive than a negative effect on student enrollment in college immediately after high school. The 
results also indicate that the probability that the effect is greater than a one percentage point increase or a one 
percentage point decrease in immediate enrollment is small, and very unlikely to be larger than a two or three 
percentage point increase or decrease (see Exhibits D.4a and D.4b). Specifically, the primary analysis indicates that: 


e There is a 58 percent chance of a positive effect, and a 42 percent chance of a negative effect of the 
messaging. 


e There is a 20 percent chance of a one percentage point increase in enrollment, and a 10 percent chance of a 
one percentage point decrease in enrollment due to the messaging. 


e There is a three percent chance of a two percentage point increase in enrollment, and a one percent chance 
of a two percentage point decrease in enrollment due to the messaging. 


There is a less than 0.1 percent chance of a three percentage point increase in enrollment, and a less than 0.1 percent 
chance of a three percentage point decrease in enrollment due to the messaging.The results from the Bayesian 
analysis can be contrasted to and/or reported along with the results from the traditional, frequentist analysis based 
on the current study alone. The traditional results indicated that the estimated size of the intervention effect was a 
0.01 percentage point increase in college enrollment immediately after high school, with a p-value of 0.99 which 
indicates that the impact estimate is highly consistent with what would have been expected if college transition 
messaging had no impact on the percentage of students who enroll in college immediately after high school. 


Exhibit D.4a Results of Bayesian Analysis - Likelihoods that College Transition Messaging Had a Positive Impact on 
Immediate College Enrollment after High School 


Percent Chance that the Impact of College Transition 
Messaging Interventions Is Greater Than: 
1 2; 3 
Percentage Percentage Percentage 
Primary Approach or Sensitivity Analysis Zero Point Points Points 
Primary Approach 
e Relevant prior studies identified by literature review 
e No adjustment for potential publication bias 
e Closed-form equations used to estimate prior mean and standard 58.3% 20.0% 2.9% 0.1% 
deviation 
e Closed-form equations used to estimate posterior mean and 
standard deviation 
Sensitivity Analysis 1 
e Relevant prior studies identified in What Works Clearinghouse™ 63.1% 28.5% 71% 0.1% 
database 
| Sensitivity Analysis 2 
e Adjustment to assume published impacts are inflated by a factor 57.0% 13.8% 0.9% <0.1% 
of two 
Sensitivity Analysis 3 
. ; ; ; 50.2% 14.7% 1.8% <0.1% 
e Adjustment to assume that mean of published impacts is zero 
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Sensitivity Analysis 4 
F : F sect a 67.1% 27.1% 4.8% 
e Iterative procedure used to estimate prior distribution 
eee ; 58.9% 21.9% 3.8% 
Sensitivity Analysis 5 (Used (Used (Used 
e Iterative procedure used to estimate posterior distribution simulated simulated simulated 
data) data) data) 
Use of “Benchmark Priors” Recommended by Deke & Finucane (2020) and reported in Herrmann et al. (2019) 
Sensitivity Analysis 6 
e All studies in What Works Clearinghouse™ database of reviewed 
studies that met standards with or without reservations 53.6% 23.5% 6.2% 
Mean and standard deviation of prior distribution reported in Table 
D.1 of Herrmann et al. (2019) 
Sensitivity Analysis 7 
e “File drawer correction” as reported in Appendix D of Herrmann 
et al. (2019) 
e All studies in What Works Clearinghouse™ database of reviewed 51.0% 21.5% 5.5% 
studies that met standards with or without reservations 
Mean and standard deviation of prior distribution reported in Table 
D.1 of Herrmann et al. (2019) 


0.3% 


0.3% 
(Used 
simulated 
data) 


0.1% 


<0.1% 


Exhibit D.4b 
Immediate College Enrollment 


Results of Bayesian Analysis -Likelihoods that College Transition Messaging had a Negative Impact on 


Primary Approach 
e Relevant prior studies identified by literature review 
e No adjustment for potential publication bias 


41.7% 10.3% 1.0% 


e Closed-form equations used to estimate prior mean and standard 
deviation 

© Closed-form equations used to estimate posterior mean and 
standard deviation 

Sensitivity Analysis 1 

e Relevant prior studies identified in What Works Clearinghouse™ 
database 


Sensitivity Analysis 2 


36.9% 10.9% 1.6% 


e Adjustment to assume published impacts are inflated by a factor 43.0% 75% 0.3% 


of two 
Sensitivity Analysis 3 


jl , ; é 49.8% 
e Adjustment to assume that mean of published impacts is zero 


14.4% 1.7% 


<0.1% 


0.1% 


<0.1% 


<0.1% 


Sensitivity Analysis 4 


. ; Ru ee 32.9% 6.7% 0.5% <0.1% 
e Iterative procedure used to estimate prior distribution 
41.1% 11.0% 1.3% <0.1% 
Sensitivity Analysis 5 (Used (Used (Used (Used 
e Iterative procedure used to estimate posterior distribution simulated simulated simulated simulated 
data) data) data) data) 
Use of “Benchmark Priors” Recommended by Deke & Finucane (2020) and reported in Herrmann et al. (2019) 
Sensitivity Analysis 6 
e All studies in What Works Clearinghouse™ database of reviewed 
studies that met standards with or without reservations 46.4% 18.3% 4.3.% 0.6% 


e@ Mean and standard deviation of prior distribution reported in 
Table D.1 of Herrmann et al. (2019) 

Sensitivity Analysis 7 

e “File drawer correction” as reported in Appendix D of Herrmann 
et al. (2019) 


e All studies in What Works Clearinghouse™ database of reviewed 49.0% 20.1% 4.9% <0.1% 
studies that met standards with or without reservations 


e@ Mean and standard deviation of prior distribution reported in 
Table D.1 of Herrmann et al. (2019) 


Results from the primary approach and the sensitivity analyses were in agreement that it is unlikely that the 
transition messaging had effects greater than 1 percentage point in a positive or negative direction, and very 
unlikely that the effects were greater than 2 percentage points in either direction. But the probabilities of a greater 
than zero effect varied from 50 or 51 percent for the sensitivity analyses that most stringently adjusted for 
publication bias (Sensitivity Analyses 3 and 7) to 58 percent for the primary approach, to 67 percent for the 
approach that used the iterative procedure to estimate the prior distribution (Sensitivity Analysis 4). Perhaps these 
variations in probabilities are not very important because all analyses lead to the conclusion that the effects of the 
transition messaging on immediate college enrollment are likely to be close to zero. That is, the effects are likely to 
be at best small and positive, or at worst negative but small. 


D.6 = Technical Details about the Analysis Methods 


This section contains technical details that can be used to replicate the analyses and results presented above. Notes 
on use of the “benchmark priors” are presented at the end of this section. 


To recap: the Bayesian analysis involved three broad steps: 


Step 1: Calculate the mean and standard deviation of impact estimates from relevant prior studies. The results 
are called “the priors.” 


Step 2: Use the priors from Step 1 in combination with the results from the current study to estimate the mean 
and standard deviation of impacts in the “posterior distribution.” 36 


36 The “posterior distribution” is an estimate of a probability distribution that describes how likely it would be to observe an impact of any 
particular size within the population of impacts from text-message-based interventions targeting summer melt. The study team’s analysis 
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Step 3: Using the mean and standard deviation of the posterior distribution, calculate the probability that the 
intervention impact is greater than a particular number (for example, greater than zero or greater than 3 
percentage points). 


D.6.1 Step 1: Calculate Weighted Mean and Standard Deviation from Relevant Prior Studies (Two 
Methods) 


Mean and SD Calculated Using Closed-Form Equations - Primary Method (PriD-1) 


Let Ip;igr and SDp;i9, denote the mean and standard deviation of combined impact estimates from the relevant prior 
studies, and w; and wX°"™4"'zed denote raw and normalized weights, respectively used in the calculation of weighted 
means and standard deviations. 


Normalized 
ua ~ where J; is the itt impact estimate from Exhibit D.2 or D.3, and 


Ywili _ X 
(1) Iprior = ~ = 


Normalized 
Lwj LW; 


(2) w; =1/(SE?) where SE; is the i” standard error of the impact estimate from Exhibit D.2 or D.3, and 


(3) wNormatized — (y x w;)/Yw;, where nis the number of prior studies (e.g., n=6 in Exhibit D.2 and n=5 in 


Exhibit D.3), and 


YwNormalized (; 


-Iprior)* 
(SwNormalized) _1 


(4) SDprior = 


Exhibit D.5 shows the meta-analytic means (Ip,;,,.) and standard deviations (SDp,;,,.) of the impact estimates from 
the relevant prior studies identified using either the literature review or WWC search approach. The prior meta- 
analytic mean and standard deviation are larger using the WWC search approach than using the literature review 
approach. 


Exhibit D.5 Mean and Standard Deviation of Impact Estimates Calculated Using Closed-Form Equations for Each of 
the Two Approaches to Identifying Relevant Prior Studies 


Approach to Selecting Prior Prior Studies 
Studies Mean Impact (Ip; ior) Standard Deviation (SDp,.i,;) 
Literature review approach 0.0047 0.0147 
WWC search approach 0.0183 0.0249 


For readers interested in reproducing the results, Exhibit D.6 shows the R and SAS code to calculate Ip,;,, and 
SDpyior using the data in Exhibit D.2. 


Exhibit D.6 R and SAS Program Code Using Closed-Form Equations to Calculate Prior Mean and Standard Deviation 
of Impact Estimates from Studies Selected using Literature Review Approach 

HHEHHHHH R Code #####HHHH 

# Data from Exhibit D.2 

Li = c(0.019,0.007,0.010,0.029,0.031,0.006,-0.014,0.017) 

SE.i = c( 0.013,0.009,0.006,0.015,0.017,0.013,0.006,0.015) 


# Calculate Raw Weights 
w.i=1 / SEi4*2 


assumes that the impacts from these interventions have a normal (bell-shaped) distribution that can be characterized by a normal 
probability density function, a mean, and a standard deviation. 
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# Calculate n 
n = length(l.i) 


# Calculated Normalized Weights 
normalized.wt = (n * w.i) / sum(w.i) 


# Calculate I.Prior 
I.Prior = sum(normalized.wt * L.i) / sum(normalized.wt) 


# Calculate SD.Prior 
SD.Prior = sqrt( sum(normalized.wt * (Li - I.Prior)*2) / (sum(normalized.wt) - 1) ) 


## Show Results 
cbind(I.Prior, SD.Prior) 


#> cbind(I.Prior, SD.Prior) 
# [.Prior SD.Prior 


#[1,] 0.004723355 0.0147245 


/* RRR EKER ARR K SAS Code a aah el 


/* data from Exhibit D.2 */ 
data temp; input Y sigma ; cards; 
0.019 .013 

0.007 .009 

0.010 .006 

0.029 .015 

0.031 .017 

0.006 .013 

-0.014 .006 

0.017 .015 

/* calculate raw weights */ 
data t2; set temp; 

w_i = 1/sigma**2; 

ones=1; 

run; 


/* sum the raw weights and get "n" */ 
proc means sum; var w_i; 

output out=t3 sum= sum_w_i; 

data t4; set t3; 

rename _Freq_ =n; ones=1; drop _type_; 


/* Calculate normalized weights and look at data*/ 
data t5; merge t2 t4; by ones; 

Normalized_w_i = (n * w_i) / sum_w i; 

proc print; run; 


/* Use Proc Means to calculate IJ_Prior and SD_Prior */ 
proc means mean std vardef=wdf; var y; weight Normalized_w_i; run; 


/* Results (Mean is I_Prior, Std Dev is SD_Prior */ 
Analysis Variable : Y 


Mean Std Dev 
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Mean and SD Calculated Using Iterative Procedures - Sensitivity Analysis (PriD-2) 


As a sensitivity analysis, a package in the R software package named “RStan”?’ was used to calculate the mean and 
standard deviation of impact estimates from the prior studies. To facilitate replication of the methods, in Exhibit D.7 


the R code that was used is provided. 


A comparison of the means and standard deviations obtained using the RStan package versus those obtained from 


the closed-form equations shows that the two methods produced similar results (Exhibit D.8). 


Exhibit D.7 R Program Code Using RStan Package to Calculate Prior Mean and Standard Deviation of Impact 
Estimates from Studies Selected Using Literature Review Approach 


# set seed for replicability 
set.seed(621412) 


# Invoke the RStan package 
library("rstan") # observe startup messages 


## Options recommended in RStan documentation 
options(mc.cores = parallel::detectCores()) 
rstan_options(auto_write = TRUE) 


## Set up STAN model code 


stanmodelcode =" 

data { 

int<lower=0> J; // number of studies 

real y[J]; // estimated treatment effects 

real<lower=0> sigma|J]; // standard error of effect estimates 
, 

parameters { 

real mu; // population treatment effect 

real<lower=0> tau; // standard deviation in treatment effects 
vector|J] eta; // unscaled deviation from mu by school 


transformed parameters { 
vector|J] theta = mu + tau * eta; // study treatment effects 


model { 

target += normal_Ipdf(eta | 0, 1); // prior log-density 
target += normal_Ipdf(y | theta, sigma); // log-likelihood 
: 


# Create data set containing impact estimates and standard errors 
# from studies identified with literature review-based approach 


LitBased_dat <- list(J = 6, 
y = c(0.019,0.007,0.010,0.029,0.031,0.006 ,-0.014,0.017), 


37 For more information about RStan, see https://cran.r-project.org/web/packages/rstan/vignettes/rstan.html and https://cran.r- 


project.org/web/packages/rstan/vignettes/rstan.html#write-a-stan-program. 
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sigma = c( 0.013,0.009,0.006,0.015,0.017,0.013, 0.006,0.015)) 


### Run the model and examine results 
fit1 = stan(model_code = stanmodelcode, 
data = LitBased_dat, 

control = list(adapt_delta = 0.9999) 


) 


print(fit1, pars = c("mu", "tau"),digits_ summary = 5) 

## Output from print statement: 

Inference for Stan model: 32af4e878d1455a0a170e2073248307e. 

4 chains, each with iter=2000; warmup=1000; thin=1; 

post-warmup draws per chain=1000, total post-warmup draws=4000. 


mean se_mean sd 2.5% 25% 50% 75% 97.5% n_eff Rhat 
mu 0.00993 0.00019 0.00718 -0.00356 0.00537 0.00962 0.01407 0.02501 1475 1.00082 
tau 0.01481 0.00023 0.00734 0.00453 0.00981 0.01351 0.01814 0.03279 1063 1.00050 


Exhibit D.8 Comparison of Mean and Standard Deviation of Prior Study Impact Estimates Calculated Using Closed- 
Form-Equations and Iterative Procedure 


Prior Studies 
Approach to Selecting Priors Mean Impact (Ip; ior) Standard Deviation (SDp,.i,;) 
Primary - Closed-form equations 0.0047 0.0147 
Sensitivity - Iterative approach 0.0099 0.0148 


D.6.2 Step 2: Estimate the Posterior Distribution (Two Methods) 


Because the study team identified two methods to estimating the posterior distribution, both methods were 
implemented so that sensitivity to the method could be assessed. 


The first (primary) method uses closed-form equations that require only summary statistics (specifically, the impact 
estimate and its standard error) from the current study and the mean and standard deviation estimated from prior 
studies in Step 1. The second method (sensitivity) requires student-level data from the current study and needs to 
be implemented using iterative procedures. The second method was implemented using the Bayesian procedures 
built into SAS’s GENMOD procedure. 


The first method was selected as primary because it has three advantages over the second method. First, it can be 
easily programmed in any standard software, or even a spreadsheet, requiring only summary statistics as the data 
inputs. Second, analysts can program the equations in their own software, and need only the data shown in this 
report to replicate the results. Third, some analysts may find it easier to understand estimation of the posterior 
distribution using the closed-form equations than estimating with a built-in procedure such GENMOD. In contrast, 
the second method might seem more like a “black box,” and it requires access to micro data (such as individual 
student records). 


Estimating Posterior Distribution Using Closed-Form Equations - Primary Method (AM-1) 


The method calcuates the mean of the posterior distribution as a weighted average of the current study’s impact 
estimate and an assumed distribution of impact estimates from prior studies. The following equation for estimating 
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the mean impact (Ip,5;,) of the posterior distribution assumes that the impacts from the current study, the prior 
studies, and the posterior distribution of impacts are all normally distributed:38 


(SDpriox)” (SDprior)? 
Ipostr = prior ( — _ = z}+ Istuay ; = 2 
(SDprior) + (SEstuay) (SDprior) + (SEseuay) 


where 


Iprior is the average of impact estimates from prior studies (calculated in Step 1) 
SDp,yior is the standard error of impact estimates from prior studies (calculated in Step 1) 
Istuay is the impact estimate from the current study 


SEstuay is the standard error of the impact estimate from the current study 


The formula indicates that if SEs¢,qy is small relative to SDp,jo,, then more weight will go to Istuay than to Ip;ior. This 
means that the larger the sample and more precisely estimated the current study’s impact is, the smaller SEgrqy will 
be and the more weight Is;,ay will have relative to Ip,io,. Similarly, if impact estimates from prior studies are based 
on small samples, or vary widely from study to study, then SDp;i9, may be large relative to SEsiyay , weighting the 
current study’s estimate (/s:1ay ) more heavily. 


The standard deviation of the posterior distribution is estimated as:39 


2 
(SDprior )* (SEssnay) 


SDpostr = fen... Weicfen. Ae 

(SDprior) + (SEstuay) 

Exhibit D.9 shows the posterior distributions obtained using in the closed-form equations for each of the two 
approaches to identifying relevant prior studies. The mean of the impacts and standard deviation of the posterior 
distribution are larger using the WWC search approach than the literature review approach. 


Exhibit D.9 Estimates of Posterior Mean and Standard Deviation from Each of the Two Approaches to Identifying 
Relevant Prior Studies 


Current Study Prior Studies Posterior Distribution 
Approach to Standard Standard Standard 
Selecting Prior Impact Error Impact Error Impact Deviation 
Studies Ustuay) (SE. Study) (prior) (SDprior) (postr) (SD postr) 
Pian 0.0001 0.0124 0.0047 0.0147 0.0020 0.0095 
review approach 
can 0.0001 0.0124 0.0183 0.0249 0.0037 0.0111 
approach 


For readers interested in reproducing the results, Exhibit D.10 shows the R and SAS code to calculate Ip,;5, and 
SDprior using the data in Exhibit D.2. 


38 ~~ This formula is documented in Michalopoulos (2012). 


39 Michalopoulos (2012). 
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Exhibit D.10 R and SAS Program Code Using Closed-Form Equations to Calculate Posterior Mean and Standard 
Deviation 

HHHHHHHH R Code #####HHHH 

#Define an R function named Posterior.fnc 

Posterior.fnc= function(I.Study, SE.Study, I.Prior, SD.Prior){ 

I.Postr = I.Prior*(1-(SD.Prior’2 / (SD.Prior’2 + SE.Study*2) ) ) + LStudy* ( (SD.Prior’2 / (SD.Prior’2 + SE.Study%2) ) ) 

SD.Postr = sqrt( (SD.Prior’2 * SE.Study*2) / (SD.Prior*2 + SE.Study’2) ) 

round(cbind(I.Study,SE.Study,I.Prior,SD.Prior,I.Postr,SD.Postr),4) 


} 


# Call the function using current study mean and SE and prior mean and SE 
# arguments 
Posterior.fnc(I.Study=.0001, SE.Study=.0124,I.Prior=0.0047234, SD.Prior=0.0147245) 


#Results 


# LStudy SE.Study I.Prior SD.Prior LPostr SD.Postr 
#[1,] 1e-04 0.0124 0.0047 0.0147 0.002 0.0095 


/* KRKKKKKREKEK KKK EK SAS Code Re ae cae 
data b1; input 

I_Study SE_Study I_Prior SD_Prior; cards; 
0.0001 0.0124 0.0047 0.0147 


data b2; set b1; 
I_Postr = I_Prior*(1-(SD_Prior**2 / (SD_Prior**2 + SE_Study**2) ) ) + IStudy* ( (SD_Prior**2 / (SD_Prior**2 + 
SE_Study**2) ) ); 
SD_Postr = sqrt( (SD_Prior**2 * SE_Study**2) / (SD_Prior**2 + SE_Study**2) ); 
run; 
proc print; run; 
f* 
Obs I_Study SE_Study I_Prior SD_Prior I_Postr SD_Postr 

1.0001 0.0124 .0047 0.0147 .002012386 .009478207 
* 

/ 


Estimating Posterior Distribution Using Iterative Procedures - Sensitivity Analysis (AM-2) 


A second method to estimating the posterior distribution uses student-level data coupled with procedures built into 
SAS’s GENMOD procedure. The GENMOD procedure fits a linear probability model of the same form as the original 
impact model used to estimate impacts of the college transition messaging on immediate enrollment (Section B.4.1). 
The “Bayes” option was applied to the model with the specifications that: 


e The “priors” are normally distributed. 
e The priors for the intervention effect, Ip,;,, and (SEp;,jo,.), were calculated as described in Step 2 above. 


e The priors for other model covariates were “flat priors.” This means that for model covariates, the prior 
means are set to zero and the prior variances are set to an arbitrarily larger number (for example, 1 
million).4° 


40 Flat priors for the other independent variables (that is, the model covariates) were used for two reasons: (1) because of lack of obvious 
alternatives—information on the prior distributions of the covariates was not available; and (2) the choice of priors for covariates is 
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Because one of the objectives for this appendix is that readers and analysts who wish to fully understand the 
approach used can replicate the results presented, and because those readers and analysts will not have access to 
the original micro-level data set with records for individual students that would be needed to implement the 
iterative approach presented in this section, code is provided to simulate data and then apply the procedure to the 
simulated data. The simulated data have the same sample size (n=2,819 students in the treatment group, n=1,984 
students in the control group) as the original data; and they have a similarly small, near zero difference between 
treatment and control group members in the immediate college enrollment outcome (denoted Y in the simulated 
data). The simulated data include two covariates (denoted X1 and X2 in the simulated data). These are included to 
show how the team coded priors for covariates in the SAS GenMod procedure. The SAS program code for simulating 
the data and using Proc GenMod to estimate the mean and standard deviation of the posterior distribution is shown 
in Exhibit D.11. 


Exhibit D.11 | SAS Program Code to Simulate Study Data and Estimate Posterior Mean and Standard Deviation Using 
Iterative Procedure 
i Bs SRR AR A A A AR A OA A AK A A A A A A A A A A A A A A A OO A A AK A A A A A RK AK A AE A OR RK KK KK KK 
Create simulated data set 
DR RAS A A A A A A A A A A A A A OK A A A A OO A A A A A A A RK A A A A OR RK KKK */ 
data temp; 
call streaminit(132); /* Setting seed for replicability */ 
/*Generate data for the Treated group (pop. percent immediately enrolling = 66.3) */ 
i=0; do while (i<2819); 
i =i+1; TrtGrp=1; Y=rand("Bernoulli",0.663); X1 = rand("Normal",100,10); X2 = rand(""Normal",0,1); 
output; end; 
/* Generate data for the control group (pop.percent immediately enrolling = 66.3)*/ 
j=0; do while (j<1984); 
j =j+1; TrtGrp=0; Y=rand("Bernoulli",0.663); X1 = rand("Normal",100,10); X2 = rand("Normal",0,1); 
output; end; 
drop ij; 
run; 


proc means n mean; class TrtGrp; run; 


/* Descriptive statistics from simulated data set 
N 

TrtGrp Obs Variable N Mean 

0 1984 Y 1984 0.6628024 

X1 1984 99.9855649 

X2 1984 -0.0177498 


1 2819 Y 2819 0.6630011 
X1 2819 99.8533178 
X2 2819 -0.0125334 


/* DIS A A A AAS 2 A A A ACA A A A A A ACA A A A A A ACA A ACA A HE A A A AA AB A A A A A AB A AK AK A A A AK IK AK KK 


Calculate the "Prior Variance" as the square of the "prior standard deviation" 
from the prior studies 


expected to be inconsequential. The current study used a random assignment design. Covariates were used to reduce residual error and 
therefore increase power to detect effects. Because the study used random assignment, it would produce unbiased estimates with or 
without covariates. Therefore the choice of priors for the covariates is expected to be inconsequential to the Bayesian analysis. 
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(literature review based approach to identifying relevant prior studies) 
DR RAR EO OE ER OE EO A KK A EO A KK OR OR KK OK KK OE ER */ 


data b1; 
input PriorMean _ PriorSD; cards; 
0.004723355 0.0147245 


data b2; set b1; 
PriorVar = PriorSD**2; 


proc print; run; /* 
Obs PriorMean PriorSD PriorVar 


1 004723355 0.014725 .000216811 


/* DRA A A A AA A 6 A A A AA A A A A A AA A A A A A AA A ASA RE A A AS A A A A ASA A 2 A A A A AK 2 OK OK 


Create a data set named "Norma|Prior" that will be used in ProcGenMod 


to specify the priors 
DRS ISR EE 2 OE ER OO A KE OO EE EO A KK KE EO OR OK OK OK KK OK OK */ 


data NormalIPrior; input 
_type_ $ Intercept TrtGrp X1 X2; datalines; 


Mean0.0 .0047233550.0 0.0 


Var 1000000 .000216811 1000000 1000000 
srun; 


/* DRA A A A AAS A A A A A AA A A A A A AA A A A A A AA A ASA HE A A AS A A A A A AA A A A A A A AB OK OK A 


Use Proc GenMod to estimate the posterior mean and standard deviation 
DRS IRR A EE 2 OE EO OE KE EO OE KE EO OK KK EE OR OR OK OK OK KK OK */ 


proc genmod data=temp; 

model Y = TrtGrp X1 X2 / dist=normal ; 

bayes seed=6752 plots=none coeffprior=normal(input=NormalPrior) ; 
run; 


/* Output from Proc GenMod 
Bayesian Analysis 
Posterior Summaries 


Standard 
Parameter N Mean Deviation 


Intercept 10000 0.6413 0.0689 
TrtGrp 10000 0.00225 0.0100 <== Posterior mean and SD 
X1 10000 0.000205 0.000683 

X2 10000 0.0109 0.00700 

Dispersion 10000 0.2237 0.00459 
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The iterative procedure was applied and the closed-form equations methods to the simulated data. The two 
approaches produced the same estimates of the mean and standard deviation of the posterior distribution (Exhibit 
D.12).44 


Exhibit D.12 Comparison of Estimates of Posterior Mean and Standard Deviation from Closed-Form Equation Method 
and Iterative Method (USING SIMULATED DATA) 


Current Study 
Approach to (Simulated Data) Prior Studies Posterior Distribution 
Estimating Standard Standard Standard 
Posterior Impact Error Impact Error Impact Deviation 
Distribution (1 Study) (SE. Study) Uprior) (SEp;ior) Upostr) (SDpostr) 
Approach 1: Closed- 
fore Sep avons ie 0.0002 0.0139 0.0047 0.0147 0.0023 0.0101 
summary of 
simulated data 
Approach 2: 
Iterative solution 
using SAS GENMOD 0.0002 0.0139 0.0047 0.0147 0.0023 0.0100 
and simulated 
student-level data 


D.6.3. Step 3: Calculate the Probability that the Intervention Impact Is Greater than a Particular 
Number 


Using the mean and standard deviation of the posterior distribution, and assuming that the posterior distribution is 
normal, the next step is to calculate the probability that college transition messaging has a positive impact on 
immediate college enrollment (or an impact larger than some particular size deemed to be meaningful). This is done 
by calculating the area under the normal curve for impacts greater than zero (or greater than a particular number). 


This calculation may be more familiar to some analysts if it is first demonstrated using a standard normal 
distribution with a mean of zero and a standard deviation of 1. With a standard normal distribution, the probability 
of a score greater than zero is 50 percent (Exhibit D.13, left panel) and the probability of a score greater than 1.96 is 
2.5 percent (Exhibit D.13, right panel). 


41 Because the iterative solution used in the second approach has a random starting point, the posterior means and standard deviations vary 
slightly over repeated analyses. Any differences in results between the iterative and closed-form approaches were at the fourth or fifth 
decimal place. 
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Exhibit D.13 Probabilities from a Standard Normal Distribution 


Standard Normal Distribution (Mean=0, SD=1) Standard Normal Distribution (Mean=0, SD=1) 


P(Score > 0) = 0.5 P(Score > 1.96) = 0.025 


ty 


Probab 


4 2 0 2 4 4 2 0 2 


Standard Dewation Standard Dewation 


Notes: In order to facilitate replication of results, examples of how to calculate the probability of a score greater than, for example, 


1.96 under a normal distribution with mean =0 and standard deviation =1 are provided as follows: 
Using R software: 1-pnorm(q=1.96,mean=0,sd=1) 
Using SAS software: DATA NORMAL; MU=0; SIGMA=1; q=1.96; Z=(q-MU)/SIGMA; PROBABILITY = 1-PROBNORM(Z); 
Using Stata software: display 1-normprob((1.96-0)/1) 


As indicated in the top row of Exhibit D.9, the posterior distribution obtained from the primary approach to the 
Bayesian analysis had a mean and standard deviation equal to Ipy,¢, = 0.0020 and SDpo.¢; = 0.0095, respectively. #2 
From that posterior distribution the probability that college transition messaging has a positive impact on 
immediate college enrollment can be calculated, and the probabilities that the impact is larger than 1, 2, or 3 
percentage points can be calculated. The results are obtained from examination of the probabilities shown in Exhibit 
D.14. For example, the top left panel of the exhibit shows that the probability that college transition messaging has 


an impact that is greater than zero is 58.3 percent. 


42 As areminder, the primary approach to the Bayesian analysis used a literature review to identify relevant prior studies, did not make an 
adjustment for potential publication bias, and used closed-form equations for calculating the prior and posterior distributions. 
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Exhibit D.14 Probabilities of Impacts that are Greater than 0, 1, 2, or 3 Percentage Points 


Posterior: Mean=0.0020, SD=0.0095 Posterior: Mean=0.0020, SD=0.0095 
P(Impact > 0.01) = 0.2 

oO So 

wt t 

8 8 
= £ 
a a 

a 8 a 8 
a a 

= = 

Oo oO 

Impact Estimates Impact Estimates 
Posterior: Mean=0.0020, SD=0.0095 Posterior: Mean=0.0020, SD=0.0095 
P(Impact > 0.03) = 0.002 

oO 

vw 

So 

oO 
= £ 
q 3 

5 8 8 
a a 

4 

oO 

-0.02 0.00 0.02 0.04 
Impact Estimates Impact Estimates 


Notes: Examples of how to calculate the probability of an impact greater than, for example, 0.01 under a normal distribution with 
mean =0.0020 and standard deviation =0.0095 are as follows: 


Using R software: 1-pnorm(q=0.01,mean=0.0020,sd=0.0095) 


Using SAS software: DATA NORMAL; MU=0.0020; SIGMA=0.0095; q=0.01; Z=(q-MU)/SIGMA; PROBABILITY = 1-PROBNORM(Z); 
Using Stata software: display 1-normprob((0.01-0.0020)/0.0095) 


D.6.4 Notes on Use of the “Benchmark Priors” 


Deke and Finucane (2020) recommended use of “benchmark priors,” and the study team obtained the relevant 
numbers from Herrmann et al. (2019, Table D.1), from the row labeled “All findings from WWC that met standards.” 
The numbers used were from the columns “Average effect size, Precision adjusted = 0.20,” “Standard deviation of 
effect sizes, Precision adjusted = 0.26,” “Average effect size, File drawer adjustment = 0.04,” and “Standard deviation 
of effect sizes, File drawer adjustment = 0.23.” The benchmark numbers presented in that table are in standard 
deviation units, whereas the Bayesian analysis presented here uses impact estimates in the metric of proportions. 
For example, the impact estimate from Linkow et al. (2020) is 0.0001 proportion units, or 0.01 percentage points. In 
order to use the benchmark numbers, they needed to be converted into proportion units or percentage points. The 
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following formulas were used to convert benchmark means and standard deviations (SDs) from standard deviation 
units into percentage points or proportion units: 


Mean in Percentage Point Units = Meanin SD Units * 100 * BaseRate * GI BaseRate) 


Mean in Proportion Units = Meanin SD Units * ,/BaseRate i (1 = BaseRate) 


SD in Percentage Point Units = SDin SD Units * 100 * Vv BaseRate * (1 a BaseRate) 
SD in Proportion Units = SDin sp units *-y BaseRate * (1 — BaseRate) 


The “BaseRate” used in these calculations was the average rate, across both the treatment and control groups 
students, of immediate college enrollment reported in the main analyses of Linkow et al. (2020). That BaseRate was 
0.663 (66.3 percent of students enrolled in college immediately after high school). For example, the benchmark 
number “Average effect size, Precision adjusted in standard deviation units = 0.20” in standard deviation units 
corresponds to “Average effect size, Precision adjusted in proportion units = 0.0945”: 


Mean in proportion units = 0-20 * x. 663 * (1 — .663) = 0.0945 


The benchmark number “Standard deviation of effect sizes, Precision adjusted in standard deviation units = 0.26” 
corresponds to “Standard deviation of effect sizes, Precision adjusted in proportion units = 0.1229.” 


Exhibit D.15 provides R and SAS program code to calculate posterior means and standard deviations, from the 
converted benchmark priors, and to estimate the probabilities that impacts are greater than zero, or greater than 1, 
2, or 3 percentage points. 


Exhibit D.15 R and SAS Program Code Using Benchmark Priors to Calculate Posterior Mean and Standard Deviation, 
and to Estimate Probabilities that Impact is Greater than Zero or 1, 2, or 3 Percentage Points 


HHEHHHHH R Code ###H###HH# 

#Define an R function named Posterior.fne 

Posterior.fnc= function(I.Study, SE.Study, I.Prior, SD.Prior){ 

I.Postr = I.Prior*(1-(SD.Prior’2 / (SD.Prior’2 + SE.Study’2) ) ) + l.Study* ( (SD.Prior*2 / (SD.Prior*2 + SE.Study%2) ) ) 
SD.Postr = sqrt( (SD.Prior’2 * SE.Study*2) / (SD.Prior*2 + SE.Study’2) ) 
round(cbind(I.Study,SE.Study,I.Prior,SD.Prior,|I.Postr,SD.Postr),4) 


} 


# Call the function using current study mean and SE and prior mean and SE 
# arguments 
Posterior.fnc(I.Study=.0001, SE.Study=.0124,I.Prior=0.0945, SD.Prior=0.1229) 


#Results 


LStudy SE.Study LPrior SD.Prior LPostr SD.Postr 
[1,] 1e-04 0.0124 0.0945 0.1229 0.0011 0.0123 


HHEHHHH HHHHHHHHH HEH HEHE 

Given these observed data, and the benchmark priors, the probability that texting intervention has an impact on on-time 
college enrollment that is greater than zero, one, two or three percentage points is calculated: 
HHTHHHHHHHHEHHEE HAH HEHE 

1-pnorm(q=0.00,mean=0.0011 ,sd= 0.0123) 

1-pnorm(q=0.01,mean=0.0011 ,sd= 0.0123) 

1-pnorm(q=0.02,mean=0.0011 ,sd= 0.0123) 

1-pnorm(q=0.03,mean=0.0011 ,sd= 0.0123) 


## Results 
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> 1-pnorm(q=0.00,mean=0.0011 ,sd= 0.0123) 
[1] 0.5356303 

> 1-pnorm(q=0.01,mean=0.0011 ,sd= 0.0123) 
[1] 0.2346627 

> 1-pnorm(q=0.02,mean=0.0011 ,sd= 0.0123) 
[1] 0.06219744 

> 1-pnorm(q=0.03,mean=0.0011 ,sd= 0.0123) 
[1] 0.009396962 


/* KKKKRKEKEKEK ERA SAS Code LEESLELEREELE EEE | 
data b1; input 

[Study SE_StudyI_Prior SD_Prior; cards; 
0.0001 0.0124 0.0945 0.1229 


data b2; set b1; 
I_Postr = I_Prior*(1-(SD_Prior**2 / (SD_Prior**2 + SE_Study**2) ) ) + I_Study* ( (SD_Prior**2 / (SD_Prior**2 + SE_Study**2) ) ); 
SD_Postr = sqrt( (SD_Prior**2 * SE_Study**2) / (SD_Prior**2 + SE_Study**2) ); 
run; 
proc print; run; 
i 
Obs I_Study SE_Study I_Prior SD_Prior _Postr SD_Postr 
1.0001 0.0124 0.0945 0.1229 00105129 0.012337 
* 
/ 


HHEHHHHHHHHEHHEH HAH HERE 
Given these observed data, and the benchmark priors, the probability that texting intervention has an impact on on-time 


college enrollment that is greater than zero, one, two or three percentage points is calculated: 
HHEHHHHAHHHE HHH HH HRHEE AHH 


DATA NORMAL; MU=0.0011; SIGMA=0.0123; q=0.00; Z=(q-MU) /SIGMA; PROBABILITY = 1-PROBNORM{(Z);proc print; run; 
DATA NORMAL; MU=0.0011; SIGMA=0.0123; q=0.01; Z=(q-MU)/SIGMA; PROBABILITY = 1-PROBNORM{(Z);proc print; run; 
DATA NORMAL; MU=0.0011; SIGMA=0.0123; q=0.02; Z=(q-MU)/SIGMA; PROBABILITY = 1-PROBNORM{(Z);proc print; run; 
DATA NORMAL; MU=0.0011; SIGMA=0.0123; q=0.03; Z=(q-MU)/SIGMA; PROBABILITY = 1-PROBNORM{(Z);proc print; run; 


Obs MU SIGMA q Z PROBABILITY 

1.0011 0.0123 0 -0.089431 0.53563 
1.0011 0.0123 0.01 0.72358 0.23466 
1.0011 0.0123 0.02 1.53659 0.062197 
1.0011 0.0123 0.03 2.34959 0.009396962 
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APPENDIX E. COLLEGE ADVISING EXPERIENCES 


The appendix provides information about students’ reported college advising experiences in the participating GEAR 
UP high schools. This detail is intended to provide information about the context in which the study’s college 
transition messaging occurred. 


The survey of students participating in GEAR UP at high schools in the study collected data on college advising 
experiences for all students who responded to the survey at the start of the study (n=9,677).*3 The survey asked 
students how often they had discussed common college advising topics with a college counselor or advisor at their 
school during their junior or senior year.*4 Exhibit E.1 presents information about how each college advising 
measure was constructed, along with the missing data rates on each of the eight college advising topics asked about 
on the survey. 


Exhibit E.1 Measures of College Advising Topics that GEAR UP Students Discussed with High School 
Counselor/Advisor 
Definition/Coding Percentage of Students 
Question 16: Since the beginning of last school year, that is, your junior year, how Missing Data (%) 
ioe many times have you discussed the following topics with a college 
College Advising counselor/advisor at your high school? Treatment Control 
Topic [5=5 or more times, 4=4 times, 3=3 times, 2=2 times, 1=1 time, 0=0 times] Group Group 
College outcomes College graduation rates, employment rates, and/or other student 3.41 3.23 
outcomes at different colleges) 
Admissions Admissions requirements [such as SAT/ACT scores, transcripts, and 3.55 3.07 
requirements letters of recommendation] for different types of colleges) 
Application timelines Timelines for applying to college 3.58 3.48 
How to complete the How to complete the Common Application 3.94 3.68 
Common Application 
Options for paying for Your family’s options for paying for college 3.97 3.78 
college 
Cost of college with How much you and your family will have to pay for college if you get 4.40 3.98 
financial aid financial aid) 
How to complete the How to complete the FAFSA form 3.69 3.48 
FAFSA 
College fit Colleges that would be a good fit for you based on your grades, Sollil 4.64 
resources, and interests 


Source: Student survey 2016 and 2017. 


Information about the relative frequency with which students in GEAR UP high schools participating in this study 


discussed college-related topics with their high school counselor/advisor is presented in Exhibit E.2. More than half 
of students reported doing so at least three times about timelines, admissions, cost after financial aid, outcomes, fit, 
and their family’s options for paying for college. In contrast, fewer than half of students reported talking as 
frequently with their counselor/advisor about how the FAFSA or the Common Application. 


43 Some 82 percent of high school seniors from participating schools responded to the student survey. No data are available for the survey 
non-respondents to check for potential non-response bias. However, because the response rate for the survey is higher than 80 percent, the 
study team can assume that the survey responses represent the sample of high school seniors from participating schools. The student 
survey can be found at: https://www.reginfo.gov/public/do/PRAViewIC?ref nbr=201503-1850-002&icID=215771. 


44 Topics were identified based on other surveys of high school students (for example, the High School Longitudinal Survey of 2009 and the 
Consortium on Chicago School Research’s Spring 2005 Survey of Chicago Public Schools Senior Student Edition). 
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It is important to note that these results may not capture other types of preparation for college that students 
received while in high school, and that students may have discussed these and other college-related topics with 
other adults. 


Exhibit E.2 How Often GEAR UP Students from Study High Schools Discussed College-related Topics with High 
School Counselor/Advisor 


Relative Frequency 
40% 60% 80% 100% 


x 


Timelines for applying to college 


Admissions requirements 
(such as SAT/ACT scores, transcripts) 


How much you and your family will have to pay 
for college if you get financial aid 


College graduation rates, employment rates, or 
other student outcomes at different colleges 


Colleges that would be a good fit for you 
based on your grades, resources, and interests 


Your family's options for paying for college 


How to complete the FAFSA 
(Free Application for Federal Student Aid) form 


How to complete the Common Application 


®@ Zero times @ One time @ Two times © Three times ®@ Four times © Five or more times 


Notes: Percentages represent share of students, of those who responded to the survey, who said they discussed a certain topic with their high 
school college counselor a given number of times (from zero to five or more). Results shown are for all students surveyed, which include students 
not eligible for the text-message-based advising study because they did not plan to enroll in college after high school. 
Sample Sizes: 

Timelines for applying to college: 9,190 students 

Admissions requirements: 9,215 students 

How much college likely to cost with financial aid: 9,124 students 

Outcomes at different colleges: 9,233 students 

College fit: 9,063 students 

Options for paying for college: 9,181 students 

How to complete the FAFSA: 9,199 students 

How to complete the Common App: 9,180 students 
Source: Student survey 2016 and 2017. 
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